Imaging apparatus, image processing apparatus, image processing method of imaging apparatus and computer program

ABSTRACT

A document camera has an image processing apparatus, which acquires the contour of an image of an original by means of a Roberts filter. The image processing apparatus detects straight lines as candidates for forming an image of the original from the acquired contour and acquires the shape of a quadrangle of the original. The image processing apparatus determines the projection parameters showing the relationship between the shape of the image of the original and the shape of the actual original from the positions of the corners of the quadrangle and executes an operation of projection/transformation on the image of the original. The document camera outputs the image data of the image to a projector, which projects an image of the original on a screen according to the image data.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to an imaging apparatus, an image processingapparatus, an imaging method of an image processing apparatus and animage processing computer program.

2. Description of the Related Art

As a result of technological development in the field of digital camera,conventional OHPs (OverHead Projectors) for reading papers and OHPsheets by means of a line scanner and projecting images thereof havebeen and being replaced by those using a text/picture (document) camerathat comprises a digital camera.

A document camera is adapted to pickup an image of the original placedon an original table by the user by means of a camera and the data ofthe original obtained by picking up the image of the original are storedin the memory of a computer and processed so that the original isenlarged and projected on a screen by means of a projector (see, interalia, Patent Document 1: Unexamined Japanese Patent Application KOKAIPublication No. 2002-354331 (pp. 2-3, FIG. 1)).

Unlike conventional OHPs, such document cameras can pick up an image ofa three-dimensional object and project an image thereof. Thus, they arereplacing the conventional OHPs.

However, when the original is shot by a known document camera from anoblique direction, the edge of the original located close to the cameragoes out of the view angle (shooting range) of the camera. The zoommagnification of the camera has to be reduced to avoid this problem.

Then, as a result the ratio of the area of the original to the totalarea of the picked-up image is reduced even if the image is subjected tokeystone correction so that only a small image of the original isprojected and the viewers will find difficulty in viewing the projectedimage. This tendency becomes remarkable particularly when the angle ofcamera movements is large.

Additionally, if the original is warped or distorted, the picked-upimage may contain an unnecessary part. Then, if the image containingsuch an unnecessary part is processed for luminance, color differenceand so on to improve the effects of the image, the image can beimproperly processed and, if such an image is projected on a screen,again the viewers will find difficulty in viewing the projected image.

In view of the above identified problems of the prior art it istherefore the object of the present invention to provide an imagingapparatus, an image processing apparatus, an image processing method ofan imaging apparatus and an image processing computer program capable ofobtaining a clearly visible image.

SUMMARY OF THE INVENTION

In the first aspect of the present invention, the above object isachieved by providing an imaging apparatus for shooting an original, theapparatus comprising: a shape acquiring unit which acquires the contourof the image of the original obtained by shooting the original andacquires the shape of the image of the original from the acquiredcontour, a correction parameter acquiring unit which acquires the imageeffect correction parameters for correcting the image effect from theimage of the original of the shape acquired by the shape acquiring unit;a projection parameter acquiring unit which determines the projectionparameters indicating the relationship between the shape of the image ofthe original and the actual shape of the original from the shape of theimage of the original acquired by the shape acquiring unit; and an imagetransforming unit which processes the image effect for the image of theoriginal having the shape acquired by the shape acquiring unit by usingthe image effect correction parameters acquired by the correctionparameter acquiring unit and performs an image transformation of theimage of the original by using the projection parameters acquired by theprojection parameter acquiring unit.

In the second aspect of the present invention, there is provided animaging apparatus for shooting an original, the apparatus comprising: ashape acquiring unit which acquires the contour of the image of theoriginal obtained by shooting the original and acquires the shape of theimage of the original from the acquired contour, an image cutting outunit which discriminates the real original section showing the contentsof the original from the shape of the image of the original acquired bythe shape acquiring unit and cuts out the image of the discriminatedreal original section; a correction parameter acquiring unit whichacquires the image effect correction parameters for correcting the imageeffect from the image of the real original section cut out by the imagecutting out unit; and an image effect processing unit which processesthe image effect of the image of the original by using the image effectcorrection parameters acquired by the correction parameter acquiringunit.

In the third aspect of the present invention, there is provided an imageprocessing apparatus for correcting the distortion of the image of theoriginal obtained by shooting the original, the apparatus comprising: ashape acquiring unit which acquires the contour of the image of theoriginal obtained by shooting the original and acquires the shape of theimage of the original from the acquired contour; a projection parameteracquiring unit which determines the projection parameters indicating therelationship between the shape of the image of the original and theactual shape of the original from the shape of the image of the originalacquired by the shape acquiring unit; a correction parameter acquiringunit which acquires the image effect correction parameters forcorrecting the image effect from the image of the original of the shapeacquired by the shape acquiring unit, and an image transforming unitwhich processes the image effect for the image of the original havingthe shape acquired by the shape acquiring unit by using the image effectcorrection parameters acquired by the correction parameter acquiringunit and performs an image transformation of the image of the originalby using the projection parameters acquired by the projection parameteracquiring unit.

In the fourth aspect of the present invention, there is provided animage processing method of an imaging apparatus for shooting anoriginal, the method comprising: a step of acquiring the contour of theimage of the original from the image of the original picked up byshooting the original and acquiring the shape of the image of theoriginal from the acquired contour, a step of determining the projectionparameters indicating the relationship between the shape of the image ofthe original and the actual shape of the original from the acquiredshape of the image of the original; a step of acquiring the image effectcorrection parameters for correcting the image effect from the shape ofthe image of the original acquired in the shape acquiring step; and astep of processing the image effect for the image of the original havingthe acquired shape by using the acquired image effect correctionparameters and performing an image transformation of the image of theoriginal by using the acquired projection parameters.

In the fifth aspect of the invention, there is provided a storage mediumstoring a computer program adapted to cause a computer to execute: aprocedure of acquiring the contour of the image of the original from theimage of the original picked up by shooting the original and acquiringthe shape of the image of the original from the acquired contour; aprocedure of determining the projection parameters indicating therelationship between the shape of the image of the original and theactual shape of the original from the acquired shape of the image of theoriginal; a procedure of acquiring the image effect correctionparameters for correcting the image effect from the shape of the imageof the original acquired by the shape acquiring procedure; and aprocedure of processing the image effect for the image of the originalhaving the acquired shape by using the acquired image effect correctionparameters and performing an image transformation of the image of theoriginal by using the acquired projection parameters.

Thus, according to the invention, it is possible to acquire an imagethat is easy to see.

BRIEF DESCRIPTION OF THE DRAWINGS

These objects and other objects and advantages of the present inventionwill become more apparent upon reading of the following detaileddescription and the accompanying drawings in which:

FIG. 1 is a schematic diagram of Embodiment 1 of the present invention,which is an imaging/image projection apparatus;

FIG. 2 is a schematic block diagram of the document camera in FIG. 1,illustrating the configuration thereof,

FIGS. 3A and 3B are schematic illustrations of the function of the imageprocessing apparatus in FIG. 2;

FIG. 4 is a schematic illustration of the keys of the operation unit inFIG. 2;

FIG. 5 is a flowchart of the basic projection process to be executed bythe imaging/image projection apparatus of FIG. 1;

FIG. 6 is a schematic illustration of the cutting out process to beexecuted by the image processing apparatus in FIG. 2;

FIG. 7 is a schematic illustration of the basic concept of extraction ofprojection parameters and affine transformation;

FIG. 8 is a flowchart of the projection parameter extraction process tobe executed by the image processing apparatus in FIG. 2;

FIG. 9 is a flowchart of the quadrangle contour extraction process to beexecuted by the image processing apparatus in FIG. 2;

FIG. 10A is a schematic illustration of a reduced luminance image andFIG. 10B is a schematic illustration of an edge image;

FIGS. 11A and 11B are schematic illustrations of the function of aRoberts filter,

FIGS. 12A and 12B are schematic illustrations of Radon transformation;

FIGS. 13A and 13B are schematic illustrations of an operation ofacquiring data for a polar coordinate system by Radon transformation ofa line of an X, Y coordinate system;

FIG. 14 is a flowchart of the process of detecting a peak point from thedata for a polar system to be executed by the image processing apparatusin FIG. 2;

FIGS. 15A, 15B, 15C and 15D are schematic illustrations of the conceptof detecting a quadrangle from the lines extracted by detecting peakpoints;

FIG. 16 is a flowchart of the quadrangle detection process to beexecuted by the image processing apparatus in FIG. 2;

FIG. 17 is a flowchart of the quadrangle selection process to beexecuted by the image processing apparatus in FIG. 2;

FIG. 18 is a flowchart of the process of determining affine parametersfrom the corners of a quadrangle to be executed by the image processingapparatus of FIG. 2;

FIG. 19 is a schematic illustration of inverse transformation foracquiring the original image from an image obtained after projectivetransformation;

FIG. 20 is a flowchart of the image transformation process using affinetransformation to be executed by the image processing apparatus in FIG.2;

FIG. 21 is a schematic illustration of an exemplar image cut out by theimage processing apparatus in FIG. 2;

FIG. 22A is a schematic illustration of an exemplar luminance histogramand FIG. 22B is a schematic illustration of an exemplar color differencehistogram;

FIG. 23 is a flowchart of the image effect correction parameterextraction process to be executed by the image processing apparatus inFIG. 2;

FIG. 24A is a schematic illustration of the image effect process to beexecuted when the background color is white and FIG. 24B is a schematicillustration of the image effect process to be executed when thebackground color is black, whereas FIG. 24C is a schematic illustrationof the image effect process to be executed when the background color isother than white and black,

FIG. 25 is a schematic illustration of an exemplar luminancetransformation graph to be used for color adjustment;

FIGS. 26A and 26B are schematic illustrations of whitening thebackground;

FIG. 27 is a flowchart of the image effect process to be executed by theimage processing apparatus in FIG. 2;

FIG. 28 is a flowchart of the background whitening process to beexecuted by the image processing apparatus in FIG. 2;

FIGS. 29A and 29B are schematic illustrations of an exemplar image forwhich the image distortion cannot be corrected by projectivetransformation;

FIG. 30 is a schematic illustration of the corresponding relationship ofthe original image, the image obtained after projective transformationand an enlarged image obtained after projective transformation;

FIG. 31 is a former half of a flowchart of the correction adjustment andimage transformation process to be executed by the image processingapparatus in FIG. 2;

FIG. 32 is a latter half of a flowchart of the correction adjustment andimage transformation process to be executed by the image processingapparatus in FIG. 2;

FIG. 33 is a schematic illustration of Embodiment 2 of the presentinvention, which is an imaging/image projection apparatus;

FIG. 34 is a schematic block diagram of the document camera in FIG. 33,illustrating the configuration thereof;

FIG. 35 is a flowchart of the basic process to be executed by thedocument camera in FIG. 34;

FIG. 36 is a former half of a flowchart of the basic document process tobe executed by the computer in FIG. 33; and

FIG. 37 is a latter half of a flowchart of the basic document process tobe executed by the computer in FIG. 33.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Now, the present invention will be described by referring to theaccompanying drawings that illustrate preferred embodiments of theinvention, which are imaging apparatuses.

The embodiments that are described below are imaging/image projectingapparatuses adapted to project an image that corresponds to the image ofan original obtained by shooting the original.

(Embodiment 1)

FIG. 1 is a schematic diagram of Embodiment 1 of the present invention,which is an imaging/image projection apparatus.

The imaging/image projection apparatus comprises a document camera 1 anda projector 2.

The text/picture (document) camera 1 is a camera system for shooting anobject of projection, which is an original 4 and has a camera section11, a pillar 12, a base seat 13 and an operation table 14. The camerasection 11 operates to pick up an image of the original 4. A digitalcamera is used for the camera section 11. The camera section 11 ismounted on the pillar 12. The base seat 13 supports the pillar 12.

As shown in FIG. 2, the document camera 1 has an image data generatingunit 21 and a data processing unit 22. The image data generating unit 21operates for shooting the original 4 and taking in the image data of theoriginal 4.

The data processing unit 22 receives the image data taken in by theimage data generating unit 21 from the latter and processes them so asto output the processed image data to the projector 2.

The image data generating unit 21 and the data processing unit 22 may bearranged in the camera section 11 shown in FIG. 1 or alternatively theymay be arranged respectively in the camera section 11 and the operationtable 14.

The image data generating unit 21 includes an optical lens section 101and an image sensor 102.

The optical lens section 101 includes condenser lens and other lensesfor the purpose of picking up an image of the original 4.

The image sensor 102 is adapted to take in the image data obtained bydigitizing the image formed by the optical lens section 101 thatconverges beams of light. It includes a CCD (Charge Coupled Device) andother components.

The data processing unit 22 has a memory 201, a display device 202, animage processing apparatus 203, an operation unit 204, a program codememory device 205 and a CPU (Central Processing Unit) 206.

The memory 201 temporarily stores the image from the image sensor 102and the image data to be transmitted to the display device 202.

The display device 202 displays the image to be transmitted to theprojector 2.

The image processing apparatus 203 performs image processes on the imagedata temporarily stored in the memory 201 for the purpose of correctingthe distortion of the image and processing the image effect.

The image processing apparatus 203 generates an image of the original 4same as an image picked up from the right front of the original 4 asshown in FIG. 3B by correcting the distortion (keystone correction) ofthe image of the original 4 picked up from an angle that is inclined orrotated relative to the imaging plane of the camera section 11 as shownin FIG. 3A.

The image processing apparatus 203 cuts out a quadrangle from thedistorted image of the original 4 and performs an operation of inversetransformation by projection/transformation of the image of the original4 using the quadrangle in order to correct the distortion of the image.

More specifically, the image processing apparatus 203 performs thefollowing operations under the control of the CPU 206.

-   (1) Extraction of affine parameters from the image of the original 4-   (2) Image transformation using the extracted affine parameters-   (3) Extraction of image effect correction parameters relating to    luminance or color difference and image effect processing-   (4) Regulation of image transformation.

These operations will be described in greater detail hereinafter.

The operation unit 204 includes switches and keys for controlling thedocument projecting operation. As the user operates any of the keys andthe switches, the operation unit 204 transmits information on theoperation to the CPU 206.

As shown in FIG. 4, the operation unit 204 has image regulation keysincluding a upper enlarging key 211, a lower enlarging key 212, a leftenlarging key 213, a right enlarging key 214, a right rotation key 215,a left rotation key 216, an enlarging key 217, a reducing key 218 and ashooting/release key 219.

The upper enlarging key 211, the lower enlarging key 212, the leftenlarging key 213 and the right enlarging key 214 areprojection/transformation keys to be used for projection/transformation.

The upper enlarging key 211 is used to rotate the upper half of theimage around a horizontal line passing through the center of the imagetoward the viewer who compares the upper half and the lower half of theimage. The lower enlarging key 212 is used to rotate the lower half ofthe image toward the viewer around the horizontal line.

The left enlarging key 213 and the right enlarging key 214 are used toregulate the distortion of the left half and that of the right half ofthe image respectively by rotating the image around a vertical linepassing through the center of the image. The left enlarging key 213 isoperated when the left half of the image is small, whereas the rightenlarging key 214 is operated when the right half of the image is small.

The right rotation key 215 and the left rotation key 216 are rotationcorrection keys for regulating the rotation of the image. The rightrotation key 215 is used to rotate the image clockwise, whereas the leftrotation key 216 is used to rotate the image counterclockwise.

The enlarging key 217 and the reducing key 218 are keys for transformingthe image. The enlarging key 217 is used to enlarge the image, whereasthe reducing key 218 is used to reduce the image.

The memory 201 stores the predetermined ratio for enlargement andreduction. If 2 is selected for the ratio, the image processingapparatus 203 doubles the size of the image when the user depresses theenlarging key 217 once. When the user depresses the enlarging key 217for another time, the operation unit 204 and the CPU 206 control theimage processing apparatus 203 to further double the size of the imagein response to the key depression so that consequently the size of theimage is quadrupled from that of the original image. When the reducingkey 218 is depressed, the size of the enlarged image is reduced to ahalf. The operation unit 204 and the CPU 206 control the imageprocessing apparatus 203 to reduce the enlarged image.

The upper enlarging key 211, the lower enlarging key 212, the leftenlarging key 213 and the right enlarging key 214 operate as cursor keyswhen the enlarging key 217 is operated. More specifically, the upperenlarging key 211 functions as upward moving key for moving the image ofthe original 4 being displayed on the display device 202 upward when theenlarging key 217 is operated. Similarly, the lower enlarging key 212functions as downward moving key for moving the image of the original 4being displayed on the display device 202 downward when the enlargingkey 217 is operated and the left enlarging key 213 functions as leftwardmoving key for moving the image of the original 4 being displayed on thedisplay device 202 leftward when the enlarging key 217 is operated,whereas the right enlarging key 214 functions as rightward moving keyfor moving the image of the original 4 being displayed on the displaydevice 202 rightward when the enlarging key 217 is operated.

The shooting/release key 219 is the key that is operated when shootingthe original 4.

The program code memory device 205 operates to store the program thatthe CPU 206 executes. Typically, it comprises a ROM (Read Only Memory).

The CPU 206 controls the various components according to the programstored in the program code memory device 205. The memory 201 operatesalso as working memory for the CPU 206.

The CPU 206 determines if the image of the original 4, which is theshooting target (hereinafter referred to as shooting target), is movingor not and switches the imaging mode according to a flowchart, whichwill be described in greater detail hereinafter.

The imaging mode includes a movie mode and a still mode. The movie modeis selected at the time when the user places the original 4, whose imagehe or she wants to project on the base seat 13. Then, the image pickedup by the camera section 11 is projected by the projector 2.

In the movie mode, the CPU 206 controls the related components so as toproject a moving image with a resolution of VGA (Video Graphics Array)(640×480 dots) at a rate of 30 fps (frames per second). Thus, the moviemode is a mode where real time has high priority but resolution has lowpriority.

The still mode is selected after the user places the original 4. In thestill mode, the camera section 11 shoots the original 4 to obtain a highresolution image so as to project a high resolution still image. If thecamera section 11 has about 3 million pixels, the still image cut outfor projection shows a resolution of XGA (eXtended Graphics Array)(1,024×768).

The CPU 206 determines if the image of the original 4 is moving or notin order to select either the movie mode or the still mode. To do this,the CPU 206 firstly determines the quantity of image change MD that isfound between the image obtained last time and the current image. Thequantity of image change MD is the quantity that represents the extentof change that is found between the image obtained last time and theimage of the original that is currently being shot. Several methods maybe conceivable for the purpose of computing such a quantity. Forexample, the CPU 206 may determine the total sum of the absolute valuesof the differences of all the pixels from the data of the image obtainedby the last shooting operation and the data of the image obtained by thecurrent shooting operation as the quantity of image change MD.

More specifically, if the image data obtained by the last shootingoperation is Pn−1 (x,y) and the image data obtained by the currentshooting operation is Pn(x,y), where 1≦x≦640 and 1≦y≦480, the quantityof image change MD is expressed by an equation 1 shown below.$\begin{matrix}{{MD} = {\sum\limits_{x = 1}^{640}{\sum\limits_{y = 1}^{480}{{{P_{n}\left( {x,y} \right)} - {P_{n - 1}\left( {x,y} \right)}}}}}} & {{Equation}\quad 1}\end{matrix}$

However, the total sum of the absolute values of the differences of allthe pixels requires a vast volume of computation. Therefore, thequantity of image change MD may alternatively be determined by samplingpixels.

Threshold Thresh 1 and threshold Thresh 2 are defined in advance asthresholds for determining if the image moves or not. In other words,the quantity of image change MD is compared with these thresholds. Thethreshold Thresh 1 is used to determine if there is a move or not. TheCPU 206 determines that there is no move if the quantity of image changeMD is smaller than the threshold Thresh 1.

The threshold Thresh 2 is used to determine if the move, if any,requires the movie mode to be selected or not. Even if there is a movesuch as a move of a shadow or that of placing a pen in the still mode,it may not be necessary to select the movie mode provided that the moveis smaller than the threshold Thresh 2.

If the quantity of image change MD is smaller than the threshold Thresh2, the CPU 206 determines that it represents only a slight move andhence it is not necessary to select the movie mode. The threshold Thresh2 is defined to be higher than the threshold Thresh 1 (Thresh 1<Thresh2). The memory 201 stores the thresholds Thresh 1 and Thresh 2 inadvance.

When the CPU 206 switches from the movie mode to the still mode becausethere is not any move, it waits for a predetermined period of time. Morespecifically, the CPU 206 measures the stationary time Ptime after itdetermines that there is not any move. The memory 201 stores apredetermined period of time Holdtime that is compared with thestationary time Ptime.

The projector 2 forms an image of the original 4 by projecting lightonto a screen 3. The document camera 1 and the projector 2 are connectedto each other typically by way of an RGB cable 15. The document camera 1outputs the image data of the original 4 taken by the camera section 11to the projector 2 by way of the cable 15.

Now, the operation of the imaging/image projection apparatus ofEmbodiment 1 will be described below.

As the user turns on the power source of the imaging/image projectionapparatus, the CPU 206 of the document camera 1 reads out the programcode from the program code memory device 205 and executes the basicprojection process.

The basic projection process will be described by referring to theflowchart of FIG. 5.

Firstly, the CPU 206 initializes the camera defining parametersincluding those of the focal point the exposure, the white balance ofthe camera section 11 of the document camera 11 (step S11).

Then, the CPU 206 initializes the imaging mode to the movie mode (stepS12). More specifically, the CPU 206 selects the movie mode for thespecified region on the memory 201 in order to initialize the imagingmode. Then, the CPU 206 so controls the operation that the image dataread out from the image sensor 102 of the camera section 11 may be usedas those of VGA.

As a result, the scene being shot by the camera section 11 istransmitted to the image sensor 102 by way of the optical lens section101 as converged light and the image sensor 102 prepares a lowresolution digital image as movie on the basis of the image of theconverged light. Then, the image sensor 102 transmits the prepareddigital image periodically to the memory 201 typically at a rate of 30frames per second.

Then, the CPU 206 resets the stationary time Ptime (step S13).

Thereafter, the CPU 206 controls the image sensor 102 and the memory 201so as to transfer the image data for the low resolution digital image tothe memory 201 from the image sensor 102 (step S14). Note that only theimage data is transferred to the memory 201 and no image is displayed onthe display device 202. No image is displayed on the display device 202because the image data for displaying the image on the display device202 is stored in a region indicated by a different address on the memory201.

Then, the CPU 206 determines the quantity of image change MD between theimage obtained last time and the current image according to the equation1 (step S15).

Subsequently, the CPU 206 determines if the imaging mode is the moviemode or the still mode (step S16).

In the initial state, the movie mode is selected for the imaging mode.Therefore, the CPU 206 determines that the imaging mode is the moviemode in the initial state and copies the image data of the moving imagestored in the memory 201 to a predetermined region of the memory 201 inorder to project the picked-up moving image (step S17). As a result, thedisplay device 202 reads the image data of the picked-up moving imagefrom the memory 201 and outputs the RGB signal to the projector 2. Theprojector 2 projects the image according to the signal.

The CPU 206 compares the above-described predetermined threshold Thresh1 and the quantity of image change MD determined at step S15 anddetermines if there is a move in the image or not on the basis of theoutcome of the comparison (step S18).

At this time, if the user is still continuing the motion of placing asheet of paper or the like, the quantity of image change MD will exceedthe threshold Thresh 1. In such a case, the CPU 206 determines thatthere is a move in the image (YES at step S18) and resets the stationarytime Ptime to read the image data for the low resolution digital image.Then, the CPU 206 determines the quantity of image change MD and writesthe data in the projection image region of the memory 201 (steps S13through S17). As a result, the imaging/image projection apparatus keepson the movie mode so that a low resolution moving image is projected onthe screen 3.

Thereafter, as the user finishes placing the sheet of paper and hencethere is no longer any move in the image, the quantity of image changeMD falls below the threshold. Then, the CPU 206 determines that there isno move in the image (NO at step S18) and adds 1 to the stationary timePtime (step S19).

Then, the CPU 206 determines if the stationary time Ptime has got to apredetermined period of time HoldTime or not (step S20).

If it is determined that the stationary time Ptime has not got to thepredetermined period of time HoldTime (NO at step S20) yet, the CPU 206reads in the image data for the low resolution digital image once againand determines the quantity of image change MD. Then, if there is nolonger any move in the image, the CPU 206 adds 1 to the stationary timePtime (steps S14 through S19). In this case, since the stationary timePtime is not reset, it is counted up by the CPU 206 each time the CPU206 reads the image data for the moving image.

If, on the other hand, it is determined that the stationary time Ptimehas got to the predetermined period of time HoldTime (YES at step S20),the CPU 206 determines that the user holds the move of the image andselects the still mode for the imaging mode (step S21).

The CPU 206 controls the image sensor 102 so as to pick up a highresolution still image (step S22). Then, the CPU 206 writes the imagedata acquired by the image sensor 102 in the memory 201. Note that,after writing the image data in the memory 201, the CPU 206 brings backthe resolution of the camera section 11 to the low resolution forimaging.

The CPU 206 controls the image processing apparatus 203 and the imageprocessing apparatus 203 extracts projection parameters necessary foroblique correction for the picked-up high resolution still image (stepS23). The projection parameters indicate the relationship between theshape of the image of the original 4 and the actual shape of theoriginal 4.

Furthermore, the image processing apparatus 203 cuts out the object ofprojection and also operates for projection transformation to acquire aproper square image according to the extracted projection parameters(step S24).

The image processing apparatus 203 extracts image effect correctionparameters that are to be used for transforming the corrected image intoa clear and easily identifiable image by performing certain processesincluding correction of the contrast (step S25).

Then, the image processing apparatus 203 performs an image effectprocess, using the extracted image effect correction parameters (stepS26).

Then, the CPU 206 writes the image data of the corrected image to apredetermined region of the memory 201 in order to project the imagedata having been subjected to image effect process as in the case of amoving image. Then, the CPU 206 outputs the image data from the displaydevice 202 to the projector 2 as RGB signal (step S27).

Once the still mode is selected, the CPU 206 resets the stationary timePtime once again to read the image data for the low resolution digitalimage and then determines the quantity of image change MD (steps S13through S15).

If it is determined here that the imaging mode is the still mode (NO atstep S16), the CPU 206 compares the determined quantity of image changeMD and the other predetermined threshold Thresh 2 (Thresh 1<Thresh 2) todetermine if there is a move in the image or not (step S28).

If the quantity of image change MD is smaller than the threshold Thresh2, the CPU 206 determines that there is not any move in the image (NO atstep S28). Then, subsequently, the CPU 206 compares the quantity ofimage change MD and the threshold Thresh 1 to determine if there is amove in the image or not (step S30).

If the quantity of image change MD is not smaller than the Thresh 2, theCPU 206 determines that there is a move in the image (YES at step S30)and selects the movie mode for the imaging mode (step S40). Then, theCPU 206 resets the stationary time Ptime (step S12).

If, on the other hand, the quantity of image change MD is smaller thanthe threshold Thresh 2 but not smaller than threshold Thresh 1, the CPU206 determines that there is a move (YES at step S30). In this case,since the quantity of image change MD is between the threshold Thresh 1and the threshold Thresh 2, the CPU 206 picks up a high resolution stillimage once again without selecting the movie mode for the imaging modeand the image processing apparatus 203 performs an image process on theimage data of the acquire high resolution still image (steps S22 throughS27).

If the quantity of image change MD is smaller than the threshold Thresh2 and also smaller than the threshold Thresh 1, the CPU 206 determinesthat the stationary state is continuing (NO at step S30). In this case,the CPU 206 determines if any of the image regulation keys is operatedor not according to the operation information output from the operationunit 204 to obtain command information for regulating the image effect(step S31).

If it is determined that none of the image regulation keys is operated(NO at step S31), the CPU 206 returns to step S13 to rest the stationarytime Ptime.

If, on the other hand, any of the image regulation keys is operated (YESat step S31), the CPU 206 performs the specified regulating operationfor the purpose of image transformation, which may be enlarging and/orrotating the image (step S32).

Then, the CPU 206 determines if the image transformation is projectiontransformation of the image, rotating the image or the like or enlargingthe image, reducing the image, moving the image or the like (step S33).

If the CPU determines that the image transformation is projectiontransformation of the image, rotating the image or the like, the imageprocessing apparatus 203 extracts image effect correction parametersonce again under the control of the CPU 206 (step S25) and performs animage effect process (step S26). If, on the other hand, the CPU 206determines that the image transformation is enlarging the image,reducing the image, moving the image or the like, the image processingapparatus 203 performs an image effect process, using the image effectcorrection parameters that are extracted from the image last time (stepS26).

In this way, the CPU 206 controls the operation of projecting the image,while selecting either the movie mode or the still mode. As a result, solong as the user is operating the imaging/image projection apparatus,priority is given to the frame frequency for image projection. On theother hand, once the image becomes stationary, the original 4, which isthe shooting target, is cut out and an image effect process is performedto project a high resolution image.

Now, the image process of the image processing apparatus 203 will bedescribed below.

Firstly, the basic concept of affine transformation that is used for(realizing) the image process will be described.

Affine transformation is being widely used for spatial transformation ofimages. In this embodiment, projection transformation is realized not byusing three-dimensional camera parameters but by using two-dimensionalaffine transformation. The coordinates (u, v) of a point beforetransformation is related by transformation such as move, enlargement,reduction, turn or the like and the coordinates (x,y) of the point aftertransformation is related by the formula 2 below. Affine transformationis also used for projection transformation. $\begin{matrix}{\left( {x^{\prime},y^{\prime},z^{\prime}} \right) = {\left( {u,v,1} \right)\begin{pmatrix}a_{11} & a_{12} & a_{13} \\a_{21} & a_{22} & a_{23} \\a_{31} & a_{32} & a_{33}\end{pmatrix}}} & {{Equation}\quad 2}\end{matrix}$

The ultimate coordinates (x,y) are computed by an equation 3 below.$\begin{matrix}\begin{matrix}{x = {\frac{x^{\prime}}{z^{\prime}} = \frac{{a_{11}u} + {a_{21}v} + a_{31}}{{a_{13}u} + {a_{23}v} + a_{33}}}} \\{y = {\frac{y^{\prime}}{z^{\prime}} = \frac{{a_{12}u} + {a_{22}v} + a_{32}}{{a_{13}u} + {a_{23}v} + a_{33}}}}\end{matrix} & {{Equation}\quad 3}\end{matrix}$

The equation 3 is for projection transformation. The coordinates (x,y)are reduced toward 0 according to the value of z′. In other words, theparameters contained in z′ affects the projection. The parameters area13, a23 and a33. Since the other parameters can be normalized byparameter a33, it is possible to make a33 equal to 1.

FIG. 6 shows the coordinates of each of the corners of the quadrangularoriginal 4 shown in FIG. 3A. The relationship between the quadranglepicked up by the camera section 11 and the image of the actual original4 will be described by referring to FIG. 7.

Referring to FIG. 7, the U-V-W coordinate system is thethree-dimensional coordinate system of the image that is picked up bythe camera section 11. Vector A(Au, Av, Aw) and vector B(Bu, By, Bw)show the sheet of the original 4 in the three-dimensional coordinatesystem U-V-W.

Vector S(Su, Sv, Sw) shows the distance between the origin of thethree-dimensional coordinate system U-V-W and the original 4 in thecoordinate system.

The projection screen shown in FIG. 7 is used to project an image of theoriginal 4 (and corresponds to the screen 3 in FIG. 1).

If the coordinate system on the projection screen is an x,y coordinatesystem, the image projected on the screen can be taken for the imagepicked up by the camera section 11. It is assumed here that theprojection screen is arranged vertically and separated from the originby distance f as measured on the W-axis. An arbitrarily selected pointP(u,v,w) on the sheet of the original 4 and the origin are connected bya straight line and it is assumed that the straight line intersects theprojection screen at point p(x,y) as expressed by using an X-Ycoordinate system. Then, the coordinates of the point p are expressed byan equation 4 below as a result of projection transformation.$\begin{matrix}\left\{ \begin{matrix}{x = {u\frac{f}{w}}} \\{y = {v\frac{f}{w}}}\end{matrix} \right. & {{Equation}\quad 4}\end{matrix}$

From the equation 4 and the relationship between the points P0, P1, P2,P3 and the points p0, p1, p2, p3 projected onto the projection screen asshown in FIG. 7, the relationships expressed by an equation 5 below areobtained. $\begin{matrix}\begin{matrix}\left\{ \begin{matrix}{{Su} = {{k1} \cdot {x0}}} \\{{Sv} = {{k1} \cdot {y0}}} \\{{Sw} = {{k1} \cdot f}}\end{matrix} \right. \\\left\{ \begin{matrix}{{Au} = {{k1} \cdot \left\{ {{x1} - {x0} + {\alpha \cdot {x1}}} \right\}}} \\{{Av} = {{k1} \cdot \left\{ {{y1} - {y0} + {\alpha \cdot {y1}}} \right\}}} \\{{Aw} = {{k1} \cdot \alpha \cdot f}}\end{matrix} \right. \\\left\{ \begin{matrix}{{Bu} = {{k1} \cdot \left\{ {{x3} - {x0} + {\beta \cdot {x3}}} \right\}}} \\{{Bv} = {{k1} \cdot \left\{ {{y3} - {y0} + {\beta \cdot {y3}}} \right\}}} \\{{Bw} = {{k1} \cdot \beta \cdot f}}\end{matrix} \right.\end{matrix} & {{Equation}\quad 5}\end{matrix}$where k1=Sw/f.

Then, the projection coefficients α, β are expressed by an equation 6below. $\begin{matrix}\begin{matrix}{\alpha = \frac{{\left( {{x0} - {x1} + {x2} - {x3}} \right) \cdot \left( {{y3} - {y2}} \right)} - {\left( {{x3} - {x2}} \right) \cdot \left( {{y0} - {y1} + {y2} - {y3}} \right)}}{{\left( {{x1} - {x2}} \right) \cdot \left( {{y3} - {y2}} \right)} - {\left( {{x3} - {x2}} \right)\left( {{y1} - {y2}} \right)}}} \\{\beta = \frac{{\left( {{x1} - {x2}} \right) \cdot \left( {{y0} - {y1} + {y2} - {y3}} \right)} - {\left( {{x0} - {x1} + {x2} - {x3}} \right) \cdot \left( {{y1} - {y2}} \right)}}{{\left( {{x1} - {x2}} \right) \cdot \left( {{y3} - {y2}} \right)} - {\left( {{x3} - {x2}} \right)\left( {{y1} - {y2}} \right)}}}\end{matrix} & {{Equation}\quad 6}\end{matrix}$

Now, projection transformation will be described below.

An arbitrarily selected point P on the original 4 is expressed by anequation 7 using the vectors S, A and B.P=S+m·A+n·B  Equation 7where

-   -   m: the coefficient of vector A (0≦m≦1)    -   n: the coefficient of vector B (0≦n≦1).

By substituting the equation 7 by the, equation 5 the coordinates x andy are expressed respectively by an equation 8 below.$\left\{ {\quad\begin{matrix}{x = \frac{{m \cdot \left( {{x1} - {x0} + {\alpha \cdot {x1}}} \right)} + {n \cdot \left( {{x3} - {x0} + {\beta \cdot {x3}}} \right)} + {x0}}{1 + {m \cdot \beta} + {n \cdot \alpha}}} \\{y = \frac{{m \cdot \left( {{y1} - {y0} + {\alpha \cdot {y1}}} \right)} + {n \cdot \left( {{y3} - {y0} + {\beta \cdot {y3}}} \right)} + {y0}}{1 + {m \cdot \alpha} + {n \cdot \beta}}}\end{matrix}} \right.$

By applying the above relationships, it will be seen that the coordinate(x′, y′, z′) is expressed by an equation 9 below. $\begin{matrix}{{\left( {x^{\prime},y^{\prime},z^{\prime}} \right) = \left( {m,n,1} \right)}\quad\begin{pmatrix}{{x1} - {x0} + {\alpha \cdot {x1}}} & {{y1} - {y0} + {\alpha \cdot {y1}}} & \alpha \\{{x3} - {x0} + {\beta \cdot {x3}}} & {{y3} - {y0} + {\beta \cdot {y3}}} & \beta \\{x0} & {y0} & 1\end{pmatrix}} & {{Equation}\quad 9}\end{matrix}$

By substituting the equation 9 by m, n, the corresponding point (x,y) ofthe picked-up image can be determined. While the corresponding point(x,y) may not necessarily be expressed by integers, the coordinatevalues of the point, or the pixel, can be determined by means of theimage interpolation technique or some other technique.

The image processing apparatus 203 firstly extracts projectionparameters from the picked-up image of the original 4 for performingsuch affine transformation (step S23 in FIG. 5).

Now, the operation of projection parameter extracting process that theimage processing apparatus 203 executes will be described by referringto the flowchart of FIG. 8.

The image processing apparatus 203 extracts the coordinates of each ofthe four corners (the contour of the quadrangle) of the image of theoriginal 4 (step S41). The image processing apparatus 203 extracts thecontour of the quadrangle following the flowchart illustrated in FIG. 9.

More specifically, the image processing apparatus 203 generates areduced luminance image from the input image in order to reduce thenumber of computational operations of the image processing process (stepS51).

Then, the image processing apparatus 203 generates an edge image of theoriginal 4 from the generated reduced luminance image (step S52).

Then, the image processing apparatus 203 detects the straight lineparameters contained in the edge image (step S53).

Thereafter, the image processing apparatus 203 generates quadrangles ascandidates for forming the contour of the original 4 from the detectedstraight line parameters (step S54).

The image processing apparatus 203 generates quadrangles as candidatesand gives priority to each of the generated quadrangles (step S42 inFIG. 8).

The image processing apparatus 203 selects a quadrangle according to thepriority and determines if the selected quadrangle can be extracted ornot (step S43).

If it is determined that the quadrangle can be extracted (YES at stepS43), the image processing apparatus 203 acquires the quadrangle as theshape of the image of the original 4 and computes the affine parametersfrom the vertex of the acquired quadrangle (step S44).

If, on the other hand, it is determined that the quadrangle cannot beextracted (NO at step S43), the image processing apparatus 203 acquiresthe image of the quadrangle having the largest area as image of theoriginal 4 (step S45) and computes the affine parameter from the vertexof the acquired quadrangle (step S44).

Now, the operation of the projection parameter extracting process willbe described more specifically.

FIG. 10A shows a reduced luminance image generated at step S51 by theimage processing apparatus 203 as an example. The image processingapparatus 203 generates an edge image as shown in FIG. 10B from such areduced luminance image by using an edge detecting filter referred to asRoberts filter (step S52). A Roberts filter is a filter that detects theedges of an image by weighting two groups of four neighboring pixels toacquire two filters M4 g them.

FIG. 11A illustrates the coefficients of the filter Δ1 and FIG. 11Billustrates those of the filter Δ2. The pixel value g(x,y) aftertransformation is expressed by an equation 10 below when the coefficientof the two filters Δ1, Δ2 are applied to the pixel value f(x,y) ofcertain selected coordinates (x,y). $\begin{matrix}\begin{matrix}{{g\left( {x,y} \right)} = \sqrt{\left( {\Delta\quad 1} \right)^{2} + \left( {\Delta\quad 2} \right)^{2}}} \\{{\Delta\quad 1} = {{1 \cdot {f\left( {x,y} \right)}} + {0 \cdot {f\left( {{x + 1},y} \right)}} +}} \\{{{0 \cdot f}\left( {x,{y - 1}} \right)} + {1 \cdot {f\left( {{x + 1},{y - 1}} \right)}}} \\{= {{f\left( {x,y} \right)} - {f\left( {{x + 1},{y - 1}} \right)}}} \\{{\Delta\quad 2} = {{0 \cdot {f\left( {x,y} \right)}} + {1 \cdot {f\left( {{x + 1},y} \right)}} -}} \\{{{1 \cdot f}\left( {x,{y - 1}} \right)} + {0 \cdot {f\left( {{x + 1},{y - 1}} \right)}}} \\{= {{f\left( {{x + 1},y} \right)} - {f\left( {x,{y - 1}} \right)}}}\end{matrix} & {{Equation}\quad 10}\end{matrix}$where g(x,y): pixel value of the coordinates (x,y) (aftertransformation)

-   -   f(x,y): pixel value of the coordinates (x,y) (before        transformation).

The edge image illustrated in FIG. 10B contains straight lineparameters. The image processing apparatus 203 detects the straight lineparameters by Radon transformation of the edge image (step S53).

Radon transformation is integral transformation for establishingcorrespondence of n-dimensional data to (n−1)-dimensional data. Morespecifically, as shown in FIGS. 12A, 12B, take f(x,y) as image data andconsider an r-θ coordinate system obtained by rotating from the x-ycoordinate system by angle θ. The image projection data p(r,θ) in the θdirection is defined by an equation 11 below. $\begin{matrix}\begin{matrix}{{p\left( {r,\theta} \right)} = {\int_{- \infty}^{\infty}{{f\left( {{{r\quad\cos\quad\theta} - {s\quad\sin\quad\theta}},{{r\quad\sin\quad\theta} + {s\quad\cos\quad\theta}}} \right)}{\mathbb{d}s}}}} \\{= {\int_{- \infty}^{\infty}{\int_{- \infty}^{\infty}{{f\left( {x,y} \right)}{\delta\left( {{x\quad\cos\quad\theta} + {y\quad\sin\quad\theta} - r} \right)}{\mathbb{d}x}{\mathbb{d}y}}}}}\end{matrix} & {{Equation}\quad 11}\end{matrix}$where δ( ): Dirac's delta function.

The transformation of the equation 11 is referred to as Radontransformation.

The image data f(x,y) shown in FIG. 12A is transformed into imageprojection data p(r,θ) as shown in FIG. 12B by Radon transformation.

As a result of Radon transformation, a straight line L of the x-ycoordinate system as shown in FIG. 13A is expressed by p=xcosα+ysinα ina polar coordinate system. Since all the straight line L is projected topoint p(ρ,α), it is possible to detect the straight line by detectingthe peak of p(ρ,α). The image processing apparatus 203 generates datap(r,θ) from the edge image by Radon transformation, using thisprinciple.

Then, the image processing apparatus 203 extracts the peak point fromthe generated data p(r,θ). The image processing apparatus 203 followsthe flowchart of FIG. 14 for this peak point extracting process.

The image processing apparatus 203 firstly detects the maximum valuepmax at p(r,θ) (step S61).

Then, the image processing apparatus 203 determines threshold pth=pmax*k(k is a constant value not smaller than 0 and not greater than 1) (stepS62).

Then, the image processing apparatus 203 generates a binary image B(r,θ)from p(r,θ) using pth as threshold (step S63).

Thereafter, the image processing apparatus 203 performs an imagelabeling operation for B(r,θ). The number of labels obtained at thistime is expressed by N1 (step S64).

The image processing apparatus 203 then checks r,θ at the maximum valueof p(r,θ) in each label region. Then, the image processing apparatus 203acquires each of the values as ri, θi (i=1 to N1) (step S65). Thesevalues are used as straight line parameters.

Next, the image processing apparatus 203 excludes remarkably distortedquadrangles from the candidates in order to generate quadranglecandidates (step S54) by using the detected straight line parameters.This is because it is safe to assume that a remarkably distortedquadrangle does not show the contour of the original 4 since thedocument camera 1 does not shoot an original 4 that is separated fromthe base seat 13 to a large extent.

More specifically, when four straight lines a1 through a4 are displayedon the display area of the LCD of the display device 202 along with thepoints of intersection p1 through p4 of the four straight lines as theoriginal is picked up by the camera section 11, the straight lines a1through a4 make candidates of straight lines that can form a quadrangle.

On the other hand, when the number of straight lines displayed on thedisplay area is less than four or when four straight lines are displayedin the display area but the number of points of intersection is lessthan four as shown in FIGS. 15B and 15C, the set of straight lines doesnot make a candidate set of straight lines for forming a quadrangle.

If the angles formed by the bottom line of the display area and thestraight lines a1 through a4 are θ11,θ12,θ21,θ22 respectively as shownin FIG. 15A, the angle most similar to the angle θ11 of the straightlineal is the angle θ12 of the straight line a2. Thus, the straight linea1 and the straight line a2 are determined to be those that are arrangedoppositely.

If the difference of the angles of the oppositely arranged straightlines a5 and a6 is large as shown in FIG. 15D, the set of straight linesa5 through a8 does not make a candidate set of straight lines forforming a quadrangle because the document camera 1 does not shoot theoriginal 4 when the original 4 is at a position separated from the baseseat 13 to a large extent. When the original 4 is placed on the baseseat 13 and the angles of the straight lines are measured, it ispossible to determine if the straight lines in the display area arethose formed by the edges of the original 4 or not. Thus, if the angulardifferences of two pairs of straight lines that face each other of thefour straight lines are dθ1 and dθ2, thresholds dθ1th, dθ2th are definedin advance respectively for the angular differences dθ1, dθ2 and storedin the memory 201.

From this point of view, the image processing apparatus 203 acquirescandidate quadrangles from a plurality of straight lines. Morespecifically, the image processing apparatus 203 executes the quadrangledetecting process by following the flowchart of FIG. 16.

Referring to FIG. 16, firstly the image processing apparatus 203initializes the number of candidate quadrangles Nr to 0 (step S71).

Then, the image processing apparatus 203 acquires the number of straightlines N1 that are being displayed in the display area (step S72).

Thereafter, the image processing apparatus 203 determines if theacquired number of straight lines N1 is not smaller than 4 or not (stepS73).

If it determined that the acquired number of straight lines N1 issmaller than 4 (NO at step S73), the image processing apparatus 203terminates the process. In this case, the number of candidatequadrangles Nr is equal to 0.

If, on the other hand, it is determined that the acquired number ofstraight lines N1 is not smaller than 4 (YES at step S73), the imageprocessing apparatus 203 determines if there is a set of straight linesthat form a quadrangle or not (step S74).

If there is a set of straight lines but there are not four points ofintersection of straight lines, the image processing apparatus 203determines that there is not any set of straight lines that form aquadrangle (NO at step S74) and terminates the process.

If, on the other hand, it is determined that there is a set of straightlines that form a quadrangle (YES at step S74), the image processingapparatus 203 acquires the angular differences dθ1, dθ2 of the pairs ofoppositely disposed straight lines (step S75).

Then, the image processing apparatus 203 determines if the angulardifference dθ 1 is smaller than dθ1th that is defined in advance or not(step S76).

If it is determined that the angular difference dθ1 is not smaller thanthe threshold dθ1th (NO at step S76), the image processing apparatus 203repeats the process for the next candidate of a set of straight linesfor forming a quadrangle (steps S74 through 75).

If, on the other hand, it is determined that the angular difference dθ1is smaller than the threshold dθ1th (YES at step S76), the imageprocessing apparatus 203 determines if the angular difference dθ2 issmaller than the threshold dθ2th or not (step S77).

If it is determined that the angular difference dθ2 is not smaller thanthe threshold dθ2th (NO at step S77), the image processing apparatus 203repeats the process for the next candidate of a set of straight linesfor forming a quadrangle (steps S74 through 76).

If, on the other hand, it is determined that the angular difference dθ2is smaller than the threshold dθ2th (YES at step S77), the imageprocessing apparatus 203 selects the quadrangle having four points ofintersection as candidate and increments the number of candidates Nr byone(step S78) and stores the number of candidates Nr in the memory 201.

The image processing apparatus 203 then stores the set of straight linesthat can form a quadrangle in the memory 201 (step S79).

If the image processing apparatus 203 determines that there is no longerany candidate for a set of straight lines that form a quadrangle as aresult of repeating the process from step S74 to step S79 (NO at stepS74), it terminates the quadrangle detecting operation.

Then, the image processing apparatus 203 selects the most adequatecandidate that represents the edges of the original 4 out of thecandidates of quadrangle.

Several methods are conceivable for the selection. In this embodiment,the outermost quadrangle is selected as most adequate candidate out ofthe picked-up quadrangles. The outermost quadrangle is defined as thequadrangle that is surrounded by two pairs of lines running respectivelyin parallel with the X-axis and the Y-axis, or two pairs of linesforming a rectangle, as shown in FIG. 6 and makes the area largest.

If the coordinates of the four corners of the rectangle Ri are (x0,y0),(x1,y1), (x2,y2), (x3,y3), the area of the quadrangle is expressed by anequation 12 below.Si={max(x0,x1,x2,x3)−min(x0,x1,x2,x3)}*{max(y0,y1,y2,y3)−min(y0,y1,y2,y3)}  Equation12

The image processing apparatus 203 selects the quadrangle, following theflowchart of FIG. 17.

The image processing apparatus 203 selects a candidate quadrangle out ofthe Nr candidate quadrangles (step S81).

Then, the image processing apparatus 203 determines the area of theselected quadrangle, using the equation 12 (step S82).

Then, the image processing apparatus 203 decrements the number ofcandidate quadrangles Nr by one (step S83).

The image processing apparatus 203 determines if the number ofcandidates Nr is equal to 0 or not (step S84).

If it is determined that the number of candidates Nr is not equal to 0(NO at step S84), the image processing apparatus 203 repeats the processfrom step S81 to step S83.

When it is determined that the number of candidates Nr has become equalto 0 as a result of repeating the process (YES at step S84), the imageprocessing apparatus 203 rearranges the data of the candidatequadrangles in the descending order of the determined areas Si (stepS85).

Then, the image processing apparatus 203 selects the first quadrangle asthe highest priority quadrangle showing the contour of the original. Inthis way, if there are a plurality of candidate quadrangles, theoutermost quadrangle is always selected as the highest priorityquadrangle. This is because, in the case of an original 4 containingtexts/pictures, while one or more than one quadrangles may be containedin the original 4, no quadrangle will be found outside the original 4.Additionally, in the case of a quadrangular table being shot, the usercan adjust the zoom lens and the shooting position with ease so that theuser may be able to extract the contour of the shooting target.

(1) Extraction of Affine Parameters from the Image of the Original 4.

Now, the method of determining the projection parameters (affineparameters) from the selected quadrangle will be described below.

It is possible to determine the affine parameters that are the elementsof the matrix of the equation 9 by using the coordinates of the fourcorners of the selected quadrangle (x0,y0), (x1,y1), (x2,y2), (x3,y3)and the equations 6 and 8.

The unit of the coordinate system is modified because m and n aredefined respectively to be 0≦m≦1 and 0≦n≦1. More specifically, the sizeof the image is scaled at the U-axis side and at the V-axis siderespectively by usc and vsc without shifting the center of the image. Ifthe center of the image is uc, vc, the scale conversion of the image isexpressed by an equation 13 below.u=m/usc+0.5−uc/usc v=n/vsc+0.5−vc/vsc  Equation 13

The coordinates (x′, y′, z′) of the quadrangle after the scaleconversion is expressed by an equation 14 below, using u, v.(x′,y′,z′)=(u,v,1)·Af  Equation 14where Af represents the affine transformation matrix.

The affine transformation matrix Af is expressed by an equation 15below. $\begin{matrix}{{Af} = {\begin{pmatrix}{1/u_{sc}} & 0 & 0 \\0 & {1/v_{sc}} & 0 \\{0.5 - \frac{u_{c}}{u_{sc}}} & {0.5 - \frac{v_{c}}{v_{sc}}} & 1\end{pmatrix} \cdot \quad\begin{pmatrix}{x_{1} - x_{0} + {\alpha \cdot x_{1}}} & {y_{1} - y_{0} + {\alpha \cdot y_{1}}} & \alpha \\{x_{3} - x_{0} + {\beta \cdot x_{3}}} & {y_{3} - y_{0} + {\beta \cdot y_{3}}} & \beta \\x_{0} & y_{0} & 1\end{pmatrix}}} & {{Equation}\quad 15}\end{matrix}$

Note that each of the elements of the affine transformation matrix Af isan affine parameter.

From the equations 14 and 15, the aspect ratio k of the quadrangle thatshows the relationship between the u-axis and the v-axis is expressed byan equation 16 below. $\begin{matrix}{k = {\frac{B}{A} = \frac{\sqrt{\begin{matrix}{\left( {{x3} - {x0} + {\beta \cdot {x3}}} \right)^{2} +} \\{\left( {{y3} - {y0} + {\beta \cdot {y3}}} \right)^{2} + \left( {\beta\quad f} \right)^{2}}\end{matrix}}}{\sqrt{\begin{matrix}{\left( {{x1} - {x0} + {\alpha \cdot {x1}}} \right)^{2} +} \\{\left( {{y1} - {y0} + {\alpha \cdot {y1}}} \right)^{2} + \left( {\alpha\quad f} \right)^{2}}\end{matrix}}}}} & {{Equation}\quad 16}\end{matrix}$where f: camera parameter.

In the case of a document camera 1, the focal length of the camerasection 11 is normally designed to be equal to the distance between thelens and the base seat 13 on which an original 4 is mounted. Therefore,the focal length of the camera section 11 may be made equal to thedistance between the camera section 11 and the original 4. Anyway, inthe case of a document camera 1, the focal length or the distancebetween the camera section 11 and the original 4 is known.

Note, however, in the case of an optical lens section 101 where the lensis a zoom lens, the focal length changes as a function of the extent ofzooming. Therefore, the aspect ratio k=B/A (absolute values) can becomputed when the focal lengths f are stored in a table or the like as afunction of the different extents of zooming.

When the maximum image (screen) size (umax, vmax) that can be output tothe screen 3 is given, uc and vc are expressed respectively by anequation 17 below.uc=umax/2 vc=vmax/2  Equation 17

When vmax/umax>k and vmax/umas≦k, an image having a desired aspect ratiok can be obtained by scaling the image, using equations 18 and 19respectively.usc=umax vsc=k*umax  Equation 18usc=vmax/k vsc=vmax  Equation 19

On the basis of the above-described idea, the image processing apparatus203 acquires affine parameters from the corners of the quadrangle. Thisprocess will be described by referring to the flowchart of FIG. 18.

Firstly, the image processing apparatus 203 computes the projectioncoefficients α,β by using the coordinates of the four corners of thequadrangle (x0,y0), (x1,y1), (x2,y2), (x3,y3) and the equation 6 (stepS91).

Then, the image processing apparatus 203 computes the aspect ratio k ofthe original 4, using the equation 16 (step S92).

Thereafter, the image processing apparatus 203 specifies the center (uc,vc) of the image, using the equation 17 (step S93).

Then, the image processing apparatus 203 compares the vmax/umax of thelargest image size and the aspect ratio k expressed by the equation 16(step S94).

If vmax/umax>k (YES at step S94), the image processing apparatus 203determines that largest image size at the V-axis side (longitudinalside), or vmax, is greater than the corresponding size of the image ofthe original 4 provided that the aspect ratio k is not changed. Then,the image processing apparatus 203 determines usc, vsc, using theequation 18 so as to make the largest image size agree with thecorresponding size of the image of the original 4 at the side of theU-axis side and decides the scale of the image of the original 4 at theV-axis side (step S95).

If, on the other hand, vmax/umax≦k (NO at step S94), the imageprocessing apparatus 203 determines that the largest image size at theU-axis side (transversal side), or umax, is greater than thecorresponding size of the image of the original 4 provided that theaspect ratio k is not changed. Then, the image processing apparatus 203determines usc, vsc, using the equation 19 so as to make the largestimage size agree with the corresponding size of the image of theoriginal 4 at the side of the V-axis side and decides the scale of theimage of the original 4 at the U-axis side (step S96).

Then, the image processing apparatus 203 determines the affinetransformation matrix Af, using the equation 15, from the computed usc,vsc, uc, vc and the coordinates of the four corners of the quadrangle(x0,y0), (x1,y1), (x2,y2), (x3,y3) (step S97).

Then, the image processing apparatus 203 acquires the affine parametersA, which are the elements of the affine transformation matrix Af (stepS98).

(2) Image Transformation Using the Extracted Affine Parameters

Now, the image processing method of preparing a corrected image by usingthe acquired affine parameters will be described below.

Firstly, for affine transformation, which may be projectiontransformation or some other transformation, using the affineparameters, assume that point p(x,y) of the original image correspondsto point P(u,v) of the image obtained by the transformation, which maybe projection transformation, using a transformation matrix Ap as shownin FIG. 19. Then, it is more preferable to determine the point p(x,y) ofthe original image that corresponds to the point P(u,v) of thetransformed image than to determine the point P(u,v) of the transformedimage that corresponds to the point p(x,y) of the original image.

An interpolation method that is derived from the bilinear method is usedhere to determine the coordinates of the point P of the transformedimage. The interpolation method derived from the bilinear method is amethod of detecting the coordinates of the point of an image(transformed image) that correspond to the coordinates of a point ofanother image (original image) and determining the (pixel) value of thepoint P(u,v) of the transformed image from the (pixel) values of fourperipheral points of the point (as expressed by coordinates) of theother image. With this method, the pixel value P of the point P of thetransformed image is computed by an equation 20 below.P(u,v)=(1−kx)*(1−ky)*p(X,Y)+kx*(1−ky)*p(X+1,Y)+(1−kx)*ky*p(X,Y+1)+kx*ky*p(X+1,Y+1)  Equation 20where kx: the decimal value of x,

-   -   ky: the decimal value of y,    -   X: integer part (x) and    -   Y: integer part (y)        provided that the coordinates of the point p of the other image        is expressed p(x,y).

The image processing apparatus 203 executes the process of the flowchartof FIG. 20 in order to determine the point p(x,y) of the original imagethat corresponds to the point P(u,v) of the transformed image.

Firstly, the image processing apparatus 203 initializes the pixelposition u of the transformed image to 0 (step S101).

Then, the image processing apparatus 203 initializes the pixel positionv of the transformed image to 0 (step S102).

Then, the image processing apparatus 203 substitutes the pixel position(u,v) of the transformed image, using the affine parameters A obtainedfrom the equation 15 and determines the pixel position (x,y) of theoriginal image, using the equation 3 (step S103).

Thereafter, the image processing apparatus 203 determines the pixelvalue P(u,v) by means of the bilinear method, using the equation 20,from the determined pixel position (x,y) (step S104).

Then, the image processing apparatus 203 increments the coordinate v ofthe corrected image by one (step S105).

Then, the image processing apparatus 203 compares the coordinate v ofthe corrected image and the maximum value vmax of the coordinate v anddetermines if the coordinate v of the corrected image is not smallerthan the maximum value vmax or not (step S106).

If it is determined that the coordinate v is smaller than the maximumvalue vmax (NO at step S106), the image processing apparatus 203 repeatsthe process of steps S103 through S105.

If it is determined that the coordinate v gets to the maximum value vmaxas a result of repeating the process of steps S103 through S105 (YES atstep S106), the image processing apparatus 203 increments the coordinateu of the corrected image by one (step S107).

The image processing apparatus 203 then compares the coordinate u of thecorrected image and the maximum value umax of the coordinate u anddetermines if the coordinate u of the corrected image is not smallerthan the maximum value umax or not (step S108).

If it is determined that the coordinate u is smaller than the maximumvalue umax (NO at step S108), the image processing apparatus 203 repeatsthe process of steps S102 through S107.

If it is determined that the coordinate u gets to the maximum value umaxas a result of repeating the process of steps S102 through S107 (YES atstep S108), the image processing apparatus 203 terminates the imagetransformation process.

(3) Extraction of Image Effect Correction Parameters Relating toLuminance or Color Difference and Image Effect Process

Now, the process for extracting image effect correction parameters fromthe image obtained in the above-described manner and the image effectprocess that utilize the parameters will be described below. The imageeffect process is an operation for obtaining a clearer image.

Image effect correction parameters include the maximum value, theminimum value and the peak value of a luminance histogram, the peakvalue and the average value of a color difference histogram and othervariables that are necessary for processing the image effect.

To extract image effect correction parameters, it is necessary to (3-a)cut out the image of the solid part of the original from the entireimage except the contour and (3-b) generate a luminance histogram and acolor difference histogram. Firstly, the process necessary forextracting image effect correction parameters will be described below.

(3-a) Cutting Out an Image

As shown in FIG. 21, an image of the original 4 may contain not onlysolid parts such as photographs, graphics and characters but also thebase seat 13 and the shadow of the sheet of the original 4 or the likethat have nothing to do with the contents of the original 4. Therefore,an inner frame and an outer frame of the entire image are defined andthen a real original section within the inner frame where the solidimage of the original 4 is found and a peripheral part between the innerframe and the outer frame where the shadow of the sheet of the original4 may typically be found are defined. Generally, the contour of theoriginal 4 that is found in the peripheral part is black more often thannot and there may be cases where it is preferable that the image datacontains the data for the peripheral part.

However, when a histogram is generated to include an image of theperipheral part of the original 4, there can be cases where thehistogram cannot be effectively extended.

Therefore, the image of the real original section is cut out to generatea luminance histogram and a color difference histogram only at the timeof taking out image effect correction parameters from them and theentire area of the image is subjected to an image effect process, usingthe image effect correction parameters taken out from the histograms.The parameters can be acquired more effectively in this way.

More specifically, if the image data including the peripheral part arefor M rows and N columns and the number of dots between the outer frameand the inner frame is K for both the X- and Y-directions, the pixeldata for the M to K rows that are pixel data for the real originalsection are acquired at the X-axis side, while the pixel data for the Nto K columns that are pixel data for the real original section areacquired at the Y-axis side. Then, the pixel data for the real originalsection are those of(M−2*K) rows and (N−2*K) columns.

(3-b) Generation of Luminance Histogram and Color Difference Histogram

Then, a luminance histogram and a color difference histogram aregenerated for the image of the real original section that is cut out inthis way.

The luminance histogram shows the distribution of the luminance values(Y) found in the real original section and is generated by counting thenumber of pixels of the real original section for each luminance value.FIG. 22A is a schematic illustration of an exemplar luminance histogram.In FIG. 22A, the horizontal axis represents the luminance value (Y)while the vertical axis represents the number of pixels. To correct theimage effect, it is necessary to determine the maximum value (Ymax), theminimum value (Ymin) and the peak value (Ypeak) as image effectcorrection parameters.

The maximum value is the value that shows the highest luminance valueout of the luminance values having a count number greater than apredetermined number obtained by counting the number of pixels for eachluminance value and the minimum value is the value that shows the lowestluminance value out of the luminance values having a count numbergreater than a predetermined number obtained by counting the number ofpixels for each luminance value. The peak value is the luminance valuewhere the largest count number. It is assumed that the peak valuerepresents the luminance value of the background color of the original 4that is the shooting target.

The color difference histogram shows the distribution of the colordifferences (J, V) found in the real original section and is generatedby counting the number of pixels of the real original section for eachcolor difference. FIG. 22B is a schematic illustration of an exemplarcolor difference histogram. In FIG. 22B, the horizontal axis representsthe color difference and the vertical axis represents the number ofpixels. In the case of a color difference histogram again, a peak value(Upeak, Vpeak) of color difference where the count number of pixels ismaximum appears. It is also assumed that the peak value represents thecolor difference of the background color of the original 4. To correctthe image effect, it is necessary to determine the peak value and theaverage value (Umean, Vmean) of the color difference histogram as imageeffect correction parameters. Note that the average value is the valueof the color difference showing the average count number.

To acquire a visually excellent image by correcting the image effect, itis necessary to change image effect correction method by the backgroundcolor of the original 4 because the correction effect varies as afunction of the background color of the original 4. Therefore, it isnecessary to determine the background color of the original 4. Thebackground color of the original 4 is determined from the peak value ofthe luminance histogram and that of the color difference histogram.

The background color of the original 4 is of one of three categories.The first category is white. A white board or a notebook provides awhite background color. The second category is black. A blackboardtypically provides a black background color. The third category is otherthan white and black. A magazine or a pamphlet provides such abackground color.

More specifically, the background color of the original 4 is determinedby means of the determining equations shown below.

(2-a) Requirement for White Background Color

The requirement for white background color is expressed by an equation21 below. When the requirement of the equation 21 is met, the backgroundcolor or the original 4 is determined to be white (W).requirement for white=(|Upeak|<color threshold) &(|Vpeak|<colorthreshold) &(Ypeak>white determining value)  Equation 21(2-b) Requirement for Black Background Color

The requirement for black background color is expressed by an equation22 below. When the requirement of the equation 22 is met, the backgroundcolor of the original 4 is determined to be black (b).requirement for black=(|Upeak|<color threshold) &(|Vpeak|<colorthreshold) &(Ypeak<black determining value)  Equation 22

When the requirements of the equations 21 and 22 are not met, thebackground color of the original 4 is determined to be a color (C) otherthan white and black. For example, 50 and 128 are selected for the colorthreshold and 128 is selected for the white determining threshold, while50 is selected for the black determining threshold.

On the basis of the above idea, the image processing apparatus 203extracts image effect correction parameters, following the flowchartillustrated in FIG. 23.

The image processing apparatus 203 counts the number of pixels for eachluminance (Y) value in the real original section and generates aluminance histogram as shown in FIG. 22A (step S111).

Then, the image processing apparatus 203 acquires the luminance maximumvalue (Ymax), the luminance minimum value (Ymin) and a luminance peakvalue (Ypeak) from the generated luminance histogram (step S112).

Then, the image processing apparatus 203 generates a color differencehistogram for color differences (U, V) as shown in FIG. 22B (step S113).

Thereafter, the image processing apparatus 203 determines the peakvalues (Upeak, Vpeak) of the color differences (U, V) as image effectcorrection parameters (step S114).

Then, the image processing apparatus 203 determines the average values(U mean, V mean) of the color differences (U, V) as image effectcorrection parameters (step S115).

Then, the image processing apparatus 203 determines the background colorof the original 4 from the peak values (Y peak, U peak, V peak) of theimage histograms, using the background color determining requirementequations 21 and 22 (step S116).

Then, the image processing apparatus 203 stores the image effectcorrection parameters and the data on the background color of theoriginal 4 in the memory 201 (step S1117).

Subsequently, the image processing apparatus 203 processes the imageeffect, using the image effect correction parameters that are extractedin the above-described manner (step S26 in FIG. 5).

As pointed out above, different processes need to be used for differentbackground colors in order to effectively carry out an image effectprocess.

When the background color is white as in the case of a white board or anote book, a luminance transformation as shown in FIG. 24A is carriedout. When the background color is black as in the case of a blackboard,a luminance transformation as shown in FIG. 24B is carried out. When thebackground color is other than white and black as in the case of amagazine or a pamphlet, a luminance transformation as shown in FIG. 24Cis carried out. In FIGS. 24A, 24B and 24C, the horizontal axisrepresents the input pixel value and the vertical axis represents theoutput pixel value.

When the background color is white, the angle of inclination of theluminance transformation line is changed before and after the peak valueas shown in FIG. 24A. For example, 230 is selected as predeterminedluminance value and the input luminance peak value is raised to theluminance level of 230. Then, the maximum value is brought up to themaximum luminance value. Therefore, the luminance transformation line isexpressed by two straight line segments as shown in FIG. 24A.

When the background color is black, a luminance transformation iscarried out to bring the peak value to a predetermined luminance level(20) as shown in FIG. 24B. In this case again, the luminancetransformation line is expressed by two straight line segments as shownin FIG. 24B.

When the background color is other than white and black, the part lowerthan the minimum value and the part higher than the maximum value arecut to define a luminance transformation line that is expressed by asingle line segment as in the case of ordinary extension as shown inFIG. 24C.

A transformation table of the luminance (Y) of the background and theoutput (Y′) may be prepared in advance and stored in the memory 201.Then, the output value of each pixel will be determined from the inputpixel value by referring to the prepared transformation table and theimage effect process will be executed. A light pixel becomes lighter anda dark pixel becomes darker in an image obtained by such atransformation process to broaden the luminance distribution and makethe image more visually recognizable.

There can be cases where the picked-up image is entirely turned yellow,for example, to reveal that the white balance of the digital camera isnot regulated properly. Such a color change cannot be corrected simplyby carrying out an image effect process, using the luminance histogram.

If such is the case, a color adjustment operation is conducted to obtaina desirable image. FIG. 25 is a schematic illustration of an exemplarluminance transformation graph to be used for color adjustment.

In FIG. 25, the horizontal axis represents the input color differencevalue and the vertical axis represents the output color differencevalue. For color adjustment, an operation of luminance transformation isconducted by referring to a luminance transformation graph as shown inFIG. 25 so as to make the average values (U mean, Vmean) of the colordifferences (U, V) shown in FIG. 22B equal to those of grey.

If the both color difference values U and V are equal to 0, there is nocolor in the image so that the color adjustment operation is conductedso as to make the both peak values (U peak, V peak) equal to 0. In otherwords, the color transformation line is defined by two line segments.The output value U′ for the input value U is expressed by an equation 23below. $\begin{matrix}\begin{matrix}{U^{\prime} = {\frac{128}{\left( {{Umean} + 128} \right)}*\left( {U - {Umean}} \right)}} & \left( {U < {Umean}} \right) \\{U^{\prime} = {\frac{127}{\left( {{Umean} + 127} \right)}*\left( {U - {Umean}} \right)}} & \left( {U \geqq {Umean}} \right)\end{matrix} & {{Equation}\quad 23}\end{matrix}$

A similar equation is used for the color difference V.

If the background color is not sufficiently whitened simply by extendingthe luminance in a manner as described above in the image effectprocess, the background color is whitened by making the color of thepart that is taken for the background in the image show white (Y: 255,U: 0, V: 0) or a color close to white.

FIG. 26A is a graph illustrating the ratio by which the luminance valueof a pixel is raised, using the luminance value of a pixel as reference(0). In FIG. 26A, the horizontal axis represents the luminance value andthe vertical value represents the ratio by which the luminance value israised. FIG. 26B is a graph illustrating the relationship between thecolor difference and the color difference changing ratio. In FIG. 26B,the horizontal axis represents the color difference and the verticalaxis represents the color difference changing ratio.

In FIG. 26A, C0 indicates the whitening range from 0 when the luminanceand the color difference are raised by 100% and C1 indicates the rangeof changing the raising ratio as a function of the luminance value. Asshown in FIGS. 26A and 26B, the color of each pixel whose luminancevalue (Y) is not smaller than a predetermined level (Y w) and whosedolor differences (U, V) are found within a predetermined range (C0) isregarded to be within the whitening range and its luminance value israised and its color differences are made equal to 0. The image effectprocess is very effective by whitening the background in this way whenthe background is not sufficiently whitened by simply extending theluminance.

Thus, on the basis of the above idea, the image processing apparatus 203executes the image effect process, following the flowchart of FIG. 27.

Then, the image processing apparatus 203 reads out the stored imageeffect correction parameters from the memory 201 (step S121).

The image processing apparatus 203 determines if the background color iswhite or not (step S122).

If it is determined that the background color is white (YES at stepS122), the image processing apparatus 203 operates for luminancetransformation to regulate the luminance histogram in a manner asdescribed above by referring to FIG. 24A in order to make the backgroundwhiter and more visible (step S123).

If, on the other hand, it is determined that the background color is notwhite (NO at step S122), the image processing apparatus 203 determinesif the background color is black or not (Step S124).

If it is determined that the background color is black (YES at stepS124), the image processing apparatus 203 operates for luminancetransformation to regulate the luminance histogram in a manner asdescribed above by referring to FIG. 24B (step S125).

If, on the other hand, it is determined that the background color is notblack (NO at step S124), the image processing apparatus 203 operates forluminance transformation to regulate the luminance histogram thatcorresponds to the background color of the original 4 in a manner asdescribed above by referring to FIG. 24C (step S126).

Then, the image processing apparatus 203 operates for transformation ina manner as described above by referring to FIG. 25 for color adjustmenton the regulated image (step S127).

Then, the image processing apparatus 203 executes a whitening process(S128) for the background, following the flowchart of FIG. 28. In FIG.28, i max and j max indicate the size of the x-axis and that of they-axis of the input image respectively.

More specifically, the image processing apparatus 203 initializes thecount value j to 0 (step S131).

Then, the image processing apparatus 203 initializes the count value ito 0 (step S132).

The image processing apparatus 203 determines if the luminance (Y) ofthe pixel (i,j) of the input image is not smaller than a predeterminedvalue (Y w) or not (step S133).

If it is determined that the luminance (Y) is not smaller than thepredetermined value (Y w) (YES at step S133), the image processingapparatus 203 determines if the absolute values of the color differencesU, V are smaller than the predetermined value C0 or not (step S134).

If it is determined that the absolute values of the color differences U,V are smaller than the predetermined value C0 (YES at step S134), theimage processing apparatus 203 selects 255 for the luminance value (Y)and 0 for the color differences (u, v) (S135), rewrites the value of thepixel (i,j) and increments the count value i by 1 (step S136).

If, on the other hand, it is determined that the absolute value of thecolor differences U, V are not smaller than the predetermined value C0(NO at step S134), the image processing apparatus 203 determines if theabsolute values of the color differences U, V of the pixel (i,j) of theinput image are smaller than the predetermined value C1 or not (stepS137).

If it is determined that the absolute values of the color differences U,V are smaller than the predetermined value C1 (YES at step S137), theimage processing apparatus 203 makes the luminance Y=Y+a*(255−Y) (stepS138), rewrites the value of the pixel (i,j) and increments the countvalue i by 1 (step S136).

If, on the other hand, it is determined that the absolute values of thecolor differences U, V are not smaller than the predetermined value C1(NO at step S137), the image processing apparatus 203 increments thecount value i by 1 without changing the luminance value (step S136).

Then, the image processing apparatus 203 compares the count value i withthe maximum value i max and determines if the count value i gets to themaximum value i max or not (step S139).

If it is determined that the count value i does not get to the maximumvalue i max (NO at step S139), the image processing apparatus 203repeats the process of steps S133 through S138 once again.

If it is determined that the count value i gets to the maximum value imax as a result of repeating the process of steps S133 through S138 (YESat step S139), the image processing apparatus 203 increments the countvalue j by 1 (step S140).

Then, the image processing apparatus 203 compares the count value j withthe maximum value j max and determines if the count value gets to themaximum value j max or not (step S141).

If it is determined that the count value j does not get to the maximumvalue j max (NO at step S141), the image processing apparatus 203repeats the process of steps S132 through S140 once again.

If it is determined that the count value j gets to the maximum value jmax as a result of repeating the process of steps S132 through S140 (YESat step S141), the image processing apparatus 203 terminates thebackground whitening process.

(4) Regulation of Image Transformation

Now, the operation of regulating the image, which is transformed once(step S33 in FIG. 5), will be described below.

If the coordinates of the extracted corners of a quadrangle containslight errors, there can be cases where the image projected by using theacquired affine parameters is not satisfactory as shown in FIGS. 29A and29B. In view of this problem, the imaging/image projection apparatus ofthis embodiment is adapted such that the user can regulate the obtainedprojection transformation.

As the user operates any of the keys of the operation unit 204, theoperation unit 204 transmits information on the user operation to theCPU 206 as command information in response to the operation. Then, theCPU 206 recognizes the information on the user operation and controlsthe image processing apparatus 203 according to its recognition.

When determining interpolated pixel Q(u′,v′) for the corrected image asshown in FIG. 30, it is a common practice that the interpolated pixelQ(u′,v′) is further subjected to inverse transformation Ai to determinecorrected image P(u,v) that corresponds to the interpolated pixelQ(u′,v′) and the corrected image P(u,v) is by turn subjected to inversetransformation to determine p(x,y) of the original image so as toperform an operation of pixel interpolation on the image p(x,y). Fortwo-stage image transformation such as projection transformation andenlargement transformation, a transformation matrix of twotransformations is synthetically determined and the transformations areconducted at a time relative to the original image. This is a way fordetermining an image that is faster and less subject to imagedegradation than the above-described method of forming an image bycarrying out inverse transformation twice.

The rotary inverse transformation matrix Ar for acquiring thepre-transformation image from the post-transformation image obtained byrotating the pre-transformation image by angle θ, using X-axis andY-axis as center of rotation, is expressed by an equation 24 below.$\begin{matrix}{{Ar} = \begin{bmatrix}{\cos\quad\theta} & {{- \sin}\quad\theta} & 0 \\{\sin\quad\theta} & {\cos\quad\theta} & 0 \\{{{- {Xc}}\quad\cos\quad\theta} - {{Yc}\quad\sin\quad\theta} + {Xc}} & {{{Xc}\quad\sin\quad\theta} - {{Yc}\quad\cos\quad\theta} + {Yc}} & 1\end{bmatrix}} & {{Equation}\quad 24}\end{matrix}$

The enlargement matrix Asc for acquiring the pre-transformation imagefrom the post-transformation image obtained by enlarging thepre-transformation image by Sc times, using X-axis and Y-axis as centerof enlargement, is expressed by an equation 25 below. $\begin{matrix}{{Asc} = \begin{bmatrix}{1/{Sc}} & 0 & 0 \\0 & {1/{Sc}} & 0 \\{{Xc}\left( {1 - \frac{1}{Sc}} \right)} & {{Yc}\left( {1 - \frac{1}{Sc}} \right)} & 1\end{bmatrix}} & {{Equation}\quad 25}\end{matrix}$

Note that, once an image is enlarged, sometimes it may be necessary toprocess the rounding errors, if any, when regulating the affineparameters and performing computations by using them. Therefore, whenenlarging an image, it is necessary to allow the affine parameters torestore the original size before the enlargement.

The displacement matrix As for acquiring the pre-transformation imagefrom the post-transformation image obtained by displacing thepre-transformation image by Tx and Ty in the X- and Y-directionsrespectively is expressed by an equation 26 below. $\begin{matrix}{{As} = \begin{bmatrix}1 & 0 & 0 \\0 & 1 & 0 \\{- {Tx}} & {- {Ty}} & 1\end{bmatrix}} & {{Equation}\quad 26}\end{matrix}$

The projection effect matrix Ap for acquiring the pre-transformationimage from the post-transformation image obtained by inclining thepre-transformation image by α and β in the X- and Y-directionsrespectively is expressed by an equation 27 below. $\begin{matrix}{{Ap} = \begin{bmatrix}1 & 0 & \alpha \\0 & 1 & \beta \\0 & 0 & 1\end{bmatrix}} & {{Equation}\quad 27}\end{matrix}$

The inverse transformation matrix A for two-stage inverse transformationis expressed by an equation 28 below.A=Ai(2)*Ai(1)  Equation 28

The image transformation regulating operation that is conducted on thebasis of the above-described idea will be described by referring to theflowchart of FIG. 31.

The CPU 206 determines if the enlargement ratio Zoom is equal to 1 ornot (step S151). The CPU 206 at first determines that the enlargementratio Zoom is equal to 1 because the enlargement ratio Zoom isinitialized to 1 in advance. If it is determined that the enlargementratio Zoom is equal to 1 (YES at step S151), the CPU 206 determines ifany of the upper enlarging key 211, the lower enlarging key 212, theleft enlarging key 213 and the right enlarging key 214 is depressed asone of the projection transformation keys or not (step S152).

If it is determined that one of the projection transformation keys isdepressed (YES at step S152), the CPU 206 identifies the operatedprojection transformation key.

If it is determined that the depressed projection transformation key isthe right enlarging key 214, the CPU 206 substitutes α=0.1 and β=0 ofthe projection effect matrix Ap of the equation 27 to acquire theinverse transformation matrix Ai=Ap (step S153).

If it is determined that the depressed projection transformation key isthe left enlarging key 213, the CPU 206 substitutes α=−0.1 and β=0 ofthe projection effect matrix Ap of the equation 27 to acquire theinverse transformation matrix Ai=Ap (step S154).

If it is determined that the depressed projection transformation key isthe upper enlarging key 211, the CPU 206 substitutes α=0 and β=0.1 ofthe projection effect matrix Ap of the equation 27 to acquire theinverse transformation matrix Ai=Ap (step S155).

If it is determined that the depressed projection transformation key isthe lower enlarging key 212, the CPU 206 substitutes α=0 and β=−0.1 ofthe projection effect matrix Ap of the equation 27 to acquire theinverse transformation matrix Ai=Ap (step S156).

If it is determined that none of the projection transformation keys isdepressed (NO at step S152), the CPU 206 determines if any of therotation keys is depressed or not (step S157).

If it is determined that one of the rotation keys is depressed (YES atstep S157), the CPU 206 identifies the depressed rotation key.

If it is determined that the depressed rotation key is the rightrotation key 215, the CPU 206 substitutes θ=−1 of the rotary inversetransformation matrix Ar of the equation 24 to acquire the inversetransformation matrix Ai=Ar (step S158).

If it is determined that the depressed rotation key is the left rotationkey 216, the CPU 206 substitutes θ=1 of the rotary inversetransformation matrix Ar of the equation 24 to acquire the inversetransformation matrix Ai=Ar (step S159).

If it is determined that none of the rotation keys is depressed (NO atstep S157), the CPU 206 saves the current affine parameters in thematrix Af so as to allow the affine parameters to restore the originalsize before the enlarging process (step S160).

If, on the other hand, it is determined that the enlargement ratio Zoomis not equal to 1 (NO at step S151), the CPU 206 determines if any ofthe cursor keys is depressed or not (step S161).

If it is determined that one of the cursor keys is depressed (YES atstep S161), the CPU 206 identifies the depressed cursor key.

If it is determined that the depressed cursor key is the rightwardmoving key (the enlarging key 217 and the right enlarging key 214), theCPU 206 substitutes the quantities of displacement Tx=64 and Ty=0 of theX-axis and the Y-axis respectively of the displacement matrix As of theequation 26 and acquires the inverse transformation matrix Ai=As (stepS162).

If it is determined that the depressed cursor key is the leftward movingkey (the enlarging key 217 and the left enlarging key 213), the CPU 206substitutes the quantities of displacement Tx=−64 and Ty=0 of the X-axisand the Y-axis respectively of the displacement matrix As of theequation 26 and acquires the inverse transformation matrix Ai=As (stepS163).

If it is determined that the depressed cursor key is the upward movingkey (the enlarging key 217 and the upper enlarging key 211), the CPU 206substitutes the quantities of displacement Tx 1 0=0 and Ty=64 of theX-axis and the Y-axis respectively of the displacement matrix As of theequation 26 and acquires the inverse transformation matrix Ai=As (stepS164).

If it is determined that the depressed cursor key is the downward movingkey (the enlarging key 217 and the lower enlarging key 212), the CPU 206substitutes the quantities of displacement Tx=0, Ty=−64 of the X-axisand the Y-axis respectively of the displacement matrix As of theequation 26 and acquires the inverse transformation matrix Ai=As (stepS165).

On the other hand, if it is determined that no cursor key is depressed(NO at step S161) or if it is determined that no rotation correction keyis depressed and the current affine parameters are saved in the matrixAf(step S160), the CPU 206 determines if either the enlarging key 217 orthe reducing key 218 is depressed or not (step S166).

If it is determined that neither the enlarging key 217 nor the reducingkey 218 is depressed (NO at step S166), the CPU 206 terminates the imagetransformation regulating process.

On the other hand, if it is determined that either the enlarging key 217or the reducing key 218 is depressed (YES at step S166), the CPU 206identifies the depressed key.

If it is determined that the depressed key is the enlarging key 217, theCPU 206 acquires a new enlargement ratio Zoom as enlargement ratioZoom=Zoom*Ratio (Ratio being 2, for example) (step S167) and substitutesSc=Ratio for Sc in the enlargement matrix Asc of the equation 25 toacquire the inverse transformation matrix Ai=Ap (step S168).

On the other hand, if it is determined that the depressed key is thereducing key 218, the CPU 206 acquires a new enlargement ratio Zoom asreduction ratio Zoom=Zoom/Ratio (step S169).

In the case where the reducing key 218 is depressed, the CPU 206determines if the enlargement ratio Zoom exceeds 1 or not (step S170).

If it is determined that the enlargement ratio Zoom exceeds 1 (Zoom>1)(YES at step S170), the CPU 206 substitutes Sc=1/Ratio for Sc in theenlargement matrix Asc of the equation 25 and acquires the inversetransformation matrix Ai=Ap (step S171).

If the inverse transformation matrix Ai is specified (steps S153 through156, S158, S159, S162 through S165, S168, S171), the CPU 206 determinesthe inverse transformation matrix A, using the equation 28 (step S172).

Then, the CPU 206 supplies the determined transformation matrix A to theimage processing apparatus 203 and controls the image processingapparatus 203 so as to execute an image transformation process accordingto the inverse transformation matrix A. Then, the image processingapparatus 203 executes the image transformation process by affinetransformation according to the supplied inverse transformation matrix A(step S173).

On the other hand, if it is determined that the enlargement ratio Zoomdoes not exceed 1 (Zoom≦1) (NO at step S170), the CPU 206 defines theenlargement ratio Zoom=1 (step S174).

Then, the CPU 206 restores the original transformation matrix A, usingA=Af (the affine parameters saved at step S160 before the enlargementprocess) (step S175).

Thus, the CPU 206 similarly supplies the determined inversetransformation matrix A to the image processing apparatus 203 andcontrols the image processing apparatus 203 so as to execute an imagetransformation process by affine transformation according to thesupplied inverse transformation matrix A (step S173).

The image prepared by enlargement or some other process is part of theoriginal 4. In many cases, such an enlarged original 4 that is to beused for a lecture or the like contains one or more than one graphsand/or one or more than one pictures and, if an image effect process ofextending the contrast or the like is executed by using only theenlarged part, there may be occasions where it is not possible toachieve the expected image quality. This is because the ratio of thebackground color is small if the original 4 contains one or more thanone graphs and/or one or more than one pictures.

Therefore, it is preferable to use the image effect correctionparameters obtained before such image transformation without modifyingthem when the image is to be enlarged, reduced or moved. For thisreason, if the image transformation is image enlargement, imagereduction, image displacement or the like at step S33 (imagetransformation) of FIG. 5, the CPU 206 skips the step of extractingimage effect correction parameters and the image processing apparatus203 operates for the image effect process, using the image effectcorrection parameters before the image transformation.

If, on the other hand, the image transformation is image projectiontransformation, image rotation or the like, the transformation processcan be executed without problem even when the shooting target (original4) is not cut out properly so that it is preferable to newly extractimage effect correction parameters from the newly prepared imageobtained by such image transformation. For this reason, if the imagetransformation is image projection transformation, image rotation or thelike, the CPU 206 moves to the process of extracting image effectcorrection parameters (step S25 in FIG. 5).

As described above, with this embodiment 1, the image processingapparatus 203 acquires the contour of the image of the original 4 andalso the shape of the quadrangle of the original 4 and then executes anoperation of projection transformation of the image of the original 4 bydetermining the projection parameters from the positions of the cornersof the quadrangle.

Therefore, if the original 4 that is the shooting target is aquadrangle, the embodiment 1 automatically cuts out an image of theoriginal 4 and transforms it to a proper square image. Thus, it is notnecessary to shoot the shooting target from right above and it ispossible to project the image of the original 4 in the right directionon the screen 3 regardless of the direction in which the original 4 isplaced.

Additionally, since the cut-out image is subjected to an optimal imageeffect process, it is possible to acquire a highly readable image evenif the original 4 is not illuminated.

Still additionally, when the cut-out image is corrected by rotating theimage and/or subjecting it to a projection effect process, the imageeffect correction parameters obtained at the first cutting out operationare used so that it is possible to prepare a clearly visible image withease. Thus, if the original 4 is warped or bent, the obtained image canbe regulated by processing the image with a simple operation.

Furthermore, since the cut-out image can be enlarged, reduced orsubjected to a similar operation and the final image is acquired notfrom the cut out image but by way of image transformation where theoriginal image is cut out and enlarged simultaneously, it is possible toprepare an image that is least degraded.

Finally, when an enlarged image of the cut-out image is subjected to animage effect process, the image effect correction parameters obtainedbefore the enlargement are stored and used for the image effect processso that it is possible to acquire a high quality image on a stablebasis.

(Embodiment 2)

Embodiment 2 of the present invention, which is an imaging/imageprojection apparatus, is provided with a computer, which operates forprocessing images.

FIG. 33 is a schematic illustration of Embodiment 2 of the presentinvention, which is an imaging/image projection apparatus.

The imaging/image projection apparatus of Embodiment 2 comprises acomputer 5 in addition to a document camera 1 and a projector 2.

The document camera 1 and the computer 5 are connected to each other byway of a communication cable 31 such as USB (Universal Serial Bus) andthe computer 5 and the projector 2 are connected to each other by way ofa video cable 32 such as RGB cable.

The document camera 1 of Embodiment 2 comprises an image compressor 207,an interface 208 and a data processing unit 22 in place of the imageprocessing apparatus 203 of Embodiment 1.

The image compressor 207 compresses the image data in order to reducethe amount of data to be transmitted to the computer 5. The imagecompressor 207 compresses still images, using techniques conforming tothe JPEG (Joint Photographic Expert Group) Standards or the like.

The interface 208 is used to transmit compressed image data to thecomputer 5 and receive an imaging command from the computer 5.

The CPU 206 has the functional feature of initializing the cameraspecifying parameters such as the focal point, the exposure and thewhite balance of the optical lens section 101 to the movie mode. Withthis arrangement, the scene taken by the camera section 11 is convergedto the image sensor 102 by way of the optical lens section 101 and theimage sensor 102 prepares image data for a low resolution digital imageas movie data from the image of the converged light. Then, the imagesensor 102 transmits the prepared digital image data periodically to thememory 201 typically at a rate of 30 frames per second.

The computer 5 controls the document camera 1 so as to send an imagingcommand to the document camera 1 and receives image data, which are tobe processed and transmitted to the projector 2.

The computer 5 has an interface 231, a display device 232, an imageprocessing apparatus 233, an operation unit 234, an HDD (Hard DiskDrive) 235, a CPU 236, a ROM (Read Only Memory) 237 and a RAM (RandomAccess Memory) 238.

The interface 231 is used to receive compressed image data and transitimaging commands and the like.

Like the display device 202 of Embodiment 1, the display device 232 isused to display the image to be transmitted to the projector 2.

The image processing apparatus 233 corrects the distortion of the imageof the received image data and performs image processes including imageeffect processes. In short, it operates like the image processingapparatus 203 of Embodiment 1. Additionally, the image processingapparatus 233 generates non-compression data by compression-decoding thecompressed image.

The image processing apparatus 233 may be realized by hardware or bysoftware. However, it is preferable to realize the image processingapparatus 233 by software because the functional features of the imageprocessing apparatus 233 can be updated by version ups.

Since the computer 5 is provided with the functional features of theimage processing apparatus 203 of the document camera 1 of Embodiment 1,it is no longer necessary to mount hardware for image processing on thecamera section 11 so that it is possible to use a commercially availablestandard digital camera.

The operation unit 234 has switches and keys that the user uses to inputdata and commands.

The HDD 235 stores data. The HDD 235 also stores the software forprocessing images/pictures that is installed in advance.

The CPU 236 controls the components of the computer 5 and also thedocument camera 1 by transmitting an imaging command for picking up ahigh resolution still image to the document camera 1.

The ROM 237 stores the basic program codes that the CPU 236 executes.

The RAM 238 stores data necessary for the CPU 236 to execute program.

Now, the operation of the imaging/image projection apparatus ofEmbodiment 2 will be described below by referring to the flowchart ofFIG. 35.

Firstly, the CPU 206 of the document camera 1 initializes the interface208, the image compressor 207 and the working memory in the memory 201(step S201).

Then, the CPU 206 also initializes the camera defining parametersincluding those of the focal point, the exposure, the white balance ofthe optical lens section 101 of the document camera 11 to the movie mode(step S202).

Thereafter, the CPU 206 checks the interface 208 to determine if thelatter receives an imaging command from the computer 5 or not (stepS203).

If it is determined that no imaging command is received (NO at stepS203), the CPU 206 determines if any of the image regulation keys of theoperation unit 204 is depressed or not according to the operationinformation from the operation unit 204 (step S204).

If it is determined that one of the image regulation keys is depressed(YES at step S204), the CPU 206 transmits information identifying thedepressed image regulation key to the computer 5 by way of the interface208 (step S205) and reads a low resolution image from the image sensor102 (step S206).

If, on the other hand, it is determined that none of the imageregulation keys is depressed (NO at step S204), the CPU 206 does nottransmit any identification information and reads a low resolution imagefrom the image sensor 102 (step S206).

The image compressor 207 compresses the image data that the CPU 206 readunder the control of the CPU 206 (step S207).

Then, the CPU 206 transmits the data of the low resolution imagecompressed by the image compressor 207 to the computer 5 by way of theinterface 208 typically at a rate of 30 frames per second (step S208).

Then, the CPU 206 and the image compressor 207 repeat the process ofsteps S203 through S208. In this way, the document camera 1 periodicallytransmits a low resolution image to the computer 5 unless it receives animaging command.

If it is determined that an imaging command is received (YES at stepS203), the CPU 206 selects the high resolution still mode for theimaging mode of the image sensor 102 and that of the optical lenssection 101 (step S209).

Then, the CPU 206 controls the camera section 11 so as to pick up a highresolution stationary image (step S210).

The CPU 206 transmits the image data of the image picked up by thecamera section 11 to the image compressor 207. The image compressor 207by turn compresses the received image data (step S211).

The image compressor 207 transmits the compressed image data of the highresolution stationary image to the computer 5 by way of the interface208 (step S212).

Then, the CPU 206 selects the low resolution movie mode (step S213). Itthen repeats the process of steps S203 through S208 until it receives animaging command and transmits the acquired low resolution image to thecomputer 5 by way of the interface 208.

As the document processing software installed in the HDD 235 is started,the computer 5 executes the process for the functions of the documentcamera according to the flowchart of FIGS. 35 and 36.

Firstly, the CPU 236 initializes the functions such as communication(step S221).

Then, the CPU 236 determines if it receives data or not and thendetermines the contents of the received data (step S222).

If the received data is that of a low resolution image, the imageprocessing apparatus 233 transforms the compressed data intouncompressed data by performing a compression-decoding operation on thecompressed image (step S223).

Then, as in the case of Embodiment 1, the CPU 236 computes the quantityof image change MD by means of the equation 1 (step S224).

Then, the CPU 236 determines if it is the movie mode or the still modethat is selected for the imaging mode (step S225).

Since the movie mode is selected in the initial state, the CPU 236determines that the movie mode is selected for the imaging mode (YES atstep S225). Then, the CPU 236 draws the received low resolution image(step S226). The image processing apparatus 233 outputs the data of thedrawn low resolution image to the projector 2 and the projector 2projects the image onto the screen 3.

The CPU 236 determines the quantity of image change MD between the imageand the image picked up last time by means of the equation 1 andcompares the determined quantity of image change MD with the predefinedthreshold Thresh 1 so as to determine if there is a move in the image ornot on the basis of the outcome of the comparison (step S227).

If it is determined that there is a move in the image (MD≧Thresh 1) (YESat step S227), the CPU 236 clears the stationary time Ptime (step S235)and continues the movie mode.

If, on the other hand, it is determined that there is not any move inthe image (Thresh 1>MD) (NO at step S227), the CPU 236 adds 1 to thestationary time Ptime (step S228).

Then, the CPU 236 determines if the move is stopped and thepredetermined period of time HoldTime has elapsed or not (step S229).

If it is determined that the predetermined period of time HoldTime hasnot elapsed yet (Ptime<HoldTime) (NO at step S229), the CPU 236maintains the movie mode and waits for reception of the next data (stepS222).

If, on the other hand, it is determined that the predetermined period oftime HoldTime has elapsed (Ptime≧HoldTime) (YES at step S229), the CPU236 selects the still mode for the imaging mode (step S230).

Then, the CPU 236 transmits an imaging command for picking up a highresolution still image (step S231) and waits for reception of stationaryimage data from the document camera 1 (step S222).

If data is received and it is determined that the received data is for alow resolution image (step S222), the CPU 236 decodes the low resolutionimage (step S223) and computes the quantity of image change MD (stepS224). Then, it determines if the movie mode is selected for the imagingmode or not (step S224).

If it is determined that the still mode is selected for the imaging mode(NO at step S225), the CPU 236 determines if there is a move or not bycomparing the quantity of image change MD with the two thresholds Thresh1 and Thresh 2 (steps S232, S233) as in the case of Embodiment 1.

If MD<Threshold 1 (NO at step S232 and YES at step S233), the CPU 236transmits an imaging command (step S231).

If Thresh 1≦MD≦Thresh 2 (NO at step S232 and NO at step S233), the CPU236 holds the stationary image that is currently being displayed andalso the movie mode for the imaging mode (step S234). Then, the CPU 236resets the stationary time Ptime to 0 (step S235). Thereafter, the CPU236 waits for reception of the next data (step S222).

If Thresh 2<MD (YES at step S232), the CPU 236 waits for reception ofthe next data, keeping the still mode (step S222).

If it is determined that the received data is for a high resolutionimage (step S222), the CPU 236 decodes the high resolution image (stepS236 in FIG. 37).

The image processing apparatus 233 extracts projection parameters andexecutes an image effect process (steps S237 through S240) as in thecase of Embodiment 1.

The image processing apparatus 233 draws the image and outputs it to theprojector 2 (step S241) and the CPU 236 waits for reception of the nextdata (step S222).

If it is determined that the received data relates to information on keyoperation, the CPU 236 determines if the still mode is selected for theimaging mode or not and, if it is determined that the still mode is notselected (NO at step S242), it waits for reception of the next data(step S222).

If, on the other hand, it is determined that the still mode is selected(YES at step S242), the CPU 236 regulates and converts the image (stepS243) and executes an image effect process that may vary depending onthe specific current image conversion (steps S244, S239, S240) as in thecase of Embodiment 1. The image processing apparatus 233 draws the imageand outputs it to the projector 2 (step S241). Then, the CPU 236 waitsfor reception of the next data (step S222).

As described above, Embodiment 2 comprises a computer 5 that executesimage processes. Therefore, it is not necessary to mount imageprocessing hardware on the camera section 11 and hence it is possible touse a commercially available digital camera for the imaging/imageprojection apparatus. Thus, it is possible to provide the system at lowcost.

The present invention may be embodied in various different ways. Inother words, the present invention is by no means limited to theabove-described embodiments.

For example, an image cutting out operation is conducted on the imageacquired by transforming an image of the original 4 by means ofprojection parameters when extracting image effect correction parametersin Embodiment 1. However, it is also possible to extract image effectcorrection parameters from the real original section without projectiontransformation even if the image of the original 4 is distorted.

The original 4 is assumed to have a quadrangular shape and a quadrangleis acquired from the contour of the original 4 to perform variousprocesses in the above-described embodiments. However, the shape of theoriginal 4 is not limited to quadrangle and may be pentagon or the like,although such a shape may not be realistic.

An imaging apparatus is described as imaging/image projection apparatusfor each of the above embodiments. However, an imaging apparatusaccording to the invention is not limited to such an imaging/imageprojection apparatus. For example, an imaging apparatus according to theinvention may not necessarily comprise a projector 2. Either or both ofthe above-described embodiments may be so configured as to shoot anoriginal 4 by means of a digital camera.

Programs are stored in a memory in advance in the above description ofthe embodiments. However, it may alternatively be so arranged that theprograms for causing a computer to operate as all or part of the deviceor execute the above-described processes are stored in acomputer-readable storage medium such as a flexible disk, CD-ROM(Compact Disk Read-Only Memory), DVD (Digital Versatile Disk), MO(Magneto-Optical disk) or the like, delivered and installed in acomputer so that the computer may operate as the above-described variousmeans or execute the above-described steps.

If the programs are stored in a disk device or the like of a server onthe Internet, they may be downloaded to a computer by way of a carrierwave.

Various embodiments and changes may be made thereunto without departingfrom the broad spirit and scope of the invention. The above-describedembodiments are intended to illustrate the present invention, not tolimit the scope of the present invention. The scope of the presentinvention is shown by the attached claims rather than the embodiments.Various modifications made within the meaning of an equivalent of theclaims of the invention and within the claims are to be regarded to bein the scope of the present invention.

This application is based on Japanese Patent Application No. 2003-354398filed on Oct. 14, 2003 and including specification, claims, drawings andsummary. The disclosure of the above Japanese Patent Application isincorporated herein by reference in its entirety.

1. An imaging apparatus for shooting an original, the apparatuscomprising: a shape acquiring unit which acquires the contour of theimage of the original obtained by shooting the original and acquires theshape of the image of the original from the acquired contour, acorrection parameter acquiring unit which acquires the image effectcorrection parameters for correcting the image effect from the image ofthe original of the shape acquired by the shape acquiring unit; aprojection parameter acquiring unit which determines the projectionparameters indicating the relationship between the shape of the image ofthe original and the actual shape of the original from the shape of theimage of the original acquired by the shape acquiring unit; and an imagetransforming unit which processes the image effect for the image of theoriginal having the shape acquired by the shape acquiring unit by usingthe image effect correction parameters acquired by the correctionparameter acquiring unit and performs an image transformation of theimage of the original by using the projection parameters acquired by theprojection parameter acquiring unit.
 2. An imaging apparatus forshooting an original, the apparatus comprising: a shape acquiring unitwhich acquires the contour of the image of the original obtained byshooting the original and acquires the shape of the image of theoriginal from the acquired contour, an image cutting out unit whichdiscriminates the real original section showing the contents of theoriginal from the shape of the image of the original acquired by theshape acquiring unit and cuts out the image of the discriminated realoriginal section; a correction parameter acquiring unit which acquiresthe image effect correction parameters for correcting the image effectfrom the image of the real original section cut out by the image cuttingout unit; and an image effect processing unit which processes the imageeffect of the image of the original by using the image effect correctionparameters acquired by the correction parameter acquiring unit.
 3. Theapparatus according to claim 2, wherein the image cutting out unit has aprojection parameter acquiring unit which acquires projection parametersindicating the relationship between the shape of the image of theoriginal and the shape of the actual original from the shape of theimage of the original acquired by the shape acquiring unit; and theprojection parameter acquiring unit determines the real original sectionout of the projected transformed image obtained byprojection/transformation using the projection parameters acquired bythe projection parameter acquiring unit and cuts out the image of thedetermined real original section.
 4. The apparatus according to claim 1or 3, wherein the projection parameter acquiring unit defines athree-dimensional (U, V, W) coordinate system in a space where theshooting target that is a quadrangle is found, arranges the projectionplane on which the shooting target is projected in the space, defines an(X, Y, Z) coordinate system on the projection plane and acquires aprojection/transformation equation formed by the projection parametersas shown by an equation 33 by establishing correspondence between therelation indicated by an equation 31 that is determined from therelations defined by equations 29 and 30 and theprojection/transformation equation of an equation 32 and the imagetransforming unit transforms the image of the shooting target accordingto the projection/transformation equation of the equation 33.P=S+m·A+n·B  Equation 29 where P: the coordinates (vector) of apredetermined point on the shooting target, S: the distance (vector)between the origin of the (U, V, W) coordinate system and the shootingtarget, A, B: the lengths (vector) of the sides of the object ofshooting, m: the coefficient of vector A (0≦m≦1) n: the coefficient ofvector B (0≦n≦1) $\begin{matrix}\begin{matrix}\left\{ \begin{matrix}{{Su} = {{k1} \cdot {x0}}} \\{{Sv} = {{k1} \cdot {y0}}} \\{{Sw} = {{k1} \cdot f}}\end{matrix} \right. \\\left\{ \begin{matrix}{{Au} = {{k1} \cdot \left\{ {{x1} - {x0} + {\alpha \cdot {x1}}} \right\}}} \\{{Av} = {{k1} \cdot \left\{ {{y1} - {y0} + {\alpha \cdot {y1}}} \right\}}} \\{{Aw} = {{k1} \cdot \alpha \cdot f}}\end{matrix} \right. \\\left\{ \begin{matrix}{{Bu} = {{k1} \cdot \left\{ {{x3} - {x0} + {\beta \cdot {x3}}} \right\}}} \\{{Bv} = {{k1} \cdot \left\{ {{y3} - {y0} + {\beta \cdot {y3}}} \right\}}} \\{{Bw} = {{k1} \cdot \beta \cdot f}}\end{matrix} \right. \\{{k1} = {{Sw}/f}}\end{matrix} & {{Equation}\quad 30}\end{matrix}$ where Su, Sv, Sw: the length of vector S in thethree-dimensional coordinate system (, V, W), Au, Av, Aw: the length ofvector A, Bu, By, Bw: the length of vector B, f: the focal length of thelens of the imaging section and α, β: the coefficients that correspondto vectors A and B respectively. $\begin{matrix}\left\{ {{\begin{matrix}{x = \frac{{m \cdot \left( {{x1} - {x0} + {\alpha \cdot {x1}}} \right)} + {n \cdot \left( {{x3} - {x0} + {\beta \cdot {x3}}} \right)} + {x0}}{1 + {m \cdot \beta} + {n \cdot \alpha}}} \\{y = \frac{{m \cdot \left( {{y1} - {y0} + {\alpha \cdot {y1}}} \right)} + {n \cdot \left( {{y3} - {y0} + {\beta \cdot {y3}}} \right)} + {y0}}{1 + {m \cdot \alpha} + {n \cdot \beta}}}\end{matrix}\alpha} = {{\frac{\begin{matrix}{{\left( {{x0} - {x1} + {x2} - {x3}} \right) \cdot \left( {{y3} - {y2}} \right)} -} \\{\left( {{x3} - {x2}} \right) \cdot \left( {{y0} - {y1} + {y2} - {y3}} \right)}\end{matrix}}{{\left( {{x1} - {x2}} \right) \cdot \left( {{y3} - {y2}} \right)} - {\left( {{x3} - {x2}} \right)\left( {{y1} - {y2}} \right)}}\beta} = \frac{\begin{matrix}{{\left( {{x1} - {x2}} \right) \cdot \left( {{y0} - {y1} + {y2} - {y3}} \right)} -} \\{\left( {{x0} - {x1} + {x2} - {x3}} \right) \cdot \left( {{y1} - {y2}} \right)}\end{matrix}}{{\left( {{x1} - {x2}} \right) \cdot \left( {{y3} - {y2}} \right)} - {\left( {{x3} - {x2}} \right)\left( {{y1} - {y2}} \right)}}}} \right. & {{Equation}\quad 31}\end{matrix}$ where x, y: the coordinates of each point of the image ofthe object of shooting on the projection plane andx0,x1,x2,x3,y0,y1,y2,y3: the coordinate values indicating the positionsof the corners (x0, y0), (x1, y1), (x2, y2) and (x3, y3) of the image ofthe shooting target as projected on the projection plane.$\begin{matrix}{\left( {x^{\prime},y^{\prime},z^{\prime}} \right) = {\left( {u,v,1} \right)\begin{pmatrix}a_{11} & a_{12} & a_{13} \\a_{21} & a_{22} & a_{23} \\a_{31} & a_{32} & a_{33}\end{pmatrix}}} & {{Equation}\quad 32}\end{matrix}$ where x′ y′, z′: the coordinates of each point on theimage of the shooting target on the projection plane andx0,x1,x2,x3,y0,y1,y2,y3: the coordinate values indicating the positionsof the corners (x0, y0), (x1, y1), (x2, y2) and (x3, y3) of the image ofthe shooting target as projected on the projection plane.$\begin{matrix}\begin{matrix}{\left( {x^{\prime},y^{\prime},z^{\prime}} \right) = \left( {m,n,1} \right)} \\{\begin{pmatrix}{{x1} - {x0} + {\alpha \cdot {x1}}} & {{y1} - {y0} + {\alpha \cdot {y1}}} & \alpha \\{{x3} - {x0} + {\beta \cdot {x3}}} & {{y3} - {y0} + {\beta \cdot {y3}}} & \beta \\{x0} & {y0} & 1\end{pmatrix}}\end{matrix} & {{Equation}\quad 33}\end{matrix}$
 5. The apparatus according to claim 3, wherein theprojection parameter acquiring unit defines a three-dimensional (U, V,W) coordinate system in a space where the shooting target that is aquadrangle is found, arranges the projection plane on which the shootingtarget is projected in the space, defines an (X, Y, Z) coordinate systemon the projection plane and acquires a projection/transformationequation formed by the projection parameters as shown by an equation 33by establishing correspondence between the relation indicated by anequation 31 that is determined from the relations defined by equations29 and 30 and the projection/transformation equation of an equation 32and the image transforming unit transforms the image of the shootingtarget according to the projection/transformation equation of theequation 33.P=S+m·A+n·B  Equation 29 where P: the coordinates (vector) of apredetermined point on the shooting target, S: the distance (vector)between the origin of the (U, V, W) coordinate system and the shootingtarget, A, B: the lengths (vector) of the sides of the object ofshooting, m: the coefficient of vector A (0 □ m □ 1) n: the coefficientof vector B (0□ n □ 1) $\begin{matrix}\left\{ {\begin{matrix}{{Su} = {{k1} \cdot {x0}}} \\{{Sv} = {{k1} \cdot {y0}}} \\{{Sw} = {{k1} \cdot f}}\end{matrix}\left\{ {\begin{matrix}{{Au} = {{k1} \cdot \left\{ {{x1} - {x0} + {\alpha \cdot {x1}}} \right\}}} \\{{Av} = {{k1} \cdot \left\{ {{y1} - {y0} + {\alpha \cdot {y1}}} \right\}}} \\{{Aw} = {{k1} \cdot \alpha \cdot f}}\end{matrix}\left\{ {{\begin{matrix}{{Bu} = {{k1} \cdot \left\{ {{x3} - {x0} + {\beta \cdot {x3}}} \right\}}} \\{{Bv} = {{k1} \cdot \left\{ {{y3} - {y0} + {\beta \cdot {y3}}} \right\}}} \\{{Bw} = {{k1} \cdot \beta \cdot f}}\end{matrix}{k1}} = {{Sw}/f}} \right.} \right.} \right. & {{Equation}\quad 30}\end{matrix}$ where Su, Sv, Sw: the length of vector S in the threedimensional coordinate system (U, V, W), Au, Av, Aw: the length ofvector A, Bu, By, Bw: the length of vector B, f: the focal length of thelens of the imaging section and α,β: the coefficients that correspond tovectors A and B respectively. $\begin{matrix}\left\{ {{\begin{matrix}{x = \frac{\begin{matrix}{{m \cdot \left( {{x1} - {x0} + {\alpha \cdot {x1}}} \right)} + {n \cdot}} \\{\left( {{x3} - {x0} + {\beta \cdot {x3}}} \right) + {x0}}\end{matrix}}{1 + {m \cdot \beta} + {n \cdot \alpha}}} \\{y = \frac{\begin{matrix}{{m \cdot \left( {{y1} - {y0} + {\alpha \cdot {y1}}} \right)} + {n \cdot}} \\{\left( {{y3} - {y0} + {\beta \cdot {y3}}} \right) + {y0}}\end{matrix}}{1 + {m \cdot \alpha} + {n \cdot \beta}}}\end{matrix}\alpha} = {{\frac{{\left( {{x0} - {x1} + {x2} - {x3}} \right) \cdot \left( {{y3} - {y2}} \right)} - {\left( {{x3} - {x2}} \right) \cdot \left( {{y0} - {y1} + {y2} - {y3}} \right)}}{{\left( {{x1} - {x2}} \right) \cdot \left( {{y3} - {y2}} \right)} - {\left( {{x3} - {x2}} \right)\left( {{y1} - {y2}} \right)}}\beta} = \frac{{\left( {{x1} - {x2}} \right) \cdot \left( {{y0} - {y1} + {y2} - {y3}} \right)} - {\left( {{x0} - {x1} + {x2} - {x3}} \right) \cdot \left( {{y1} - {y2}} \right)}}{{\left( {{x1} - {x2}} \right) \cdot \left( {{y3} - {y2}} \right)} - {\left( {{x3} - {x2}} \right)\left( {{y1} - {y2}} \right)}}}} \right. & {{Equation}\quad 31}\end{matrix}$ where x, y: the coordinates of each point of the image ofthe object of shooting on the projection plane and x0,x1,x2,x3, y0, y1,y2, y3: the coordinate values indicating the positions of the corners(x0, y0), (x1, y1), (x2, y2) and (x3, y3) of the image of the shootingtarget as projected on the projection plane. $\begin{matrix}{\left( {x^{\prime},y^{\prime},z^{\prime}} \right) = {\left( {u,v,1} \right)\begin{pmatrix}a_{11} & a_{12} & a_{13} \\a_{21} & a_{22} & a_{23} \\a_{31} & a_{32} & a_{33}\end{pmatrix}}} & {{Equation}\quad 32}\end{matrix}$ where x′ y′, z′: the coordinates of each point on theimage of the shooting target on the projection plane andx0,x1,x2,x3,y0,y1,y2,y3: the coordinate values indicating the positionsof the corners (x0, y0), (x1, y1), (x2,y2) and (x3, y3) of the image ofthe shooting target as projected on the projection plane.$\begin{matrix}\begin{matrix}{\left( {x^{\prime},y^{\prime},z^{\prime}} \right) = \left( {m,n,1} \right)} \\{\begin{pmatrix}{{x1} - {x0} + {\alpha \cdot {x1}}} & {{y1} - {y0} + {\alpha \cdot {y1}}} & \alpha \\{{x3} - {x0} + {\beta \cdot {x3}}} & {{y3} - {y0} + {\beta \cdot {y3}}} & \beta \\{x0} & {y0} & 1\end{pmatrix}}\end{matrix} & {{Equation}\quad 33}\end{matrix}$
 6. The apparatus according to claim 1, wherein the shapeacquiring unit has an edge image detecting unit which detects an edgeimage of the image of the original; and a straight line detecting unitwhich detects the straight lines that form candidates for contour of theoriginal from the edge image detected by the edge image detecting unit;and acquires the contour of the original by combining the straight linesdetected by the straight line detecting unit.
 7. The apparatus accordingto claim 6, wherein the shape acquiring unit acquires the shape havingthe largest area as the shape of the image of the original out of theshapes formed by combining the straight lines detected by the straightline detecting unit.
 8. The apparatus according to claim 2, wherein thecorrection parameter acquiring unit generates a luminance histogramshowing the relationship between each luminance value of the image ofthe solid image part cut out from the image and the number of pixelshaving the luminance value and acquires parameters for correcting theluminance histogram as image effect correction parameters from thegenerated luminance histogram.
 9. The apparatus according to claims 2,wherein the correction parameter acquiring unit generates a colordifference histogram showing the relationship between each colordifference of the image of the solid image part cut out from the imageand the number of pixels having the color difference and acquiresparameters for correcting the color difference histogram as image effectcorrection parameters from the generated color difference histogram. 10.The apparatus according to claim 9, wherein the image effect processingunit acquires the color difference having the largest number of pixelsas image effect correction parameter from the generated color differencehistogram and regulates the color of the image of the original in animage effect process for the image of the original.
 11. The apparatusaccording to claim 2, wherein the image effect processing unit acquirescommand information on the image effect process and executes the imageeffect process according to the acquired command information.
 12. Theapparatus according to claim 1, further comprising: an imaging sectionsupported on a support pillar and adapted to pick up an image of theoriginal.
 13. The apparatus according to claim 1, further comprising aprojecting section which projects the image obtained by imagetransformation by the image transforming unit on a screen.
 14. An imageprocessing apparatus for correcting the distortion of the image of theoriginal obtained by shooting the original; the apparatus comprising: ashape acquiring unit which acquires the contour of the image of theoriginal obtained by shooting the original and acquires the shape of theimage of the original from the acquired contour, a projection parameteracquiring unit which determines the projection parameters indicating therelationship between the shape of the image of the original and theactual shape of the original from the shape of the image of the originalacquired by the shape acquiring unit; a correction parameter acquiringunit which acquires the image effect correction parameters forcorrecting the image effect from the image of the original of the shapeacquired by the shape acquiring unit; and an image transforming unitwhich processes the image effect for the image of the original havingthe shape acquired by the shape acquiring unit by using the image effectcorrection parameters acquired by the correction parameter acquiringunit and performs an image transformation of the image of the originalby using the projection parameters acquired by the projection parameteracquiring unit.
 15. An image processing method of an imaging apparatusfor shooting an original, the method comprising; a step of acquiring thecontour of the image of the original from the image of the originalpicked up by shooting the original and acquiring the shape of the imageof the original from the acquired contour, a step of determining theprojection parameters indicating the relationship between the shape ofthe image of the original and the actual shape of the original from theacquired shape of the image of the original; a step of acquiring theimage effect correction parameters for correcting the image effect fromthe shape of the image of the original acquired in the shape acquiringstep; and a step of processing the image effect for the image of theoriginal having the acquired shape by using the acquired image effectcorrection parameters and performing an image transformation of theimage of the original.
 16. A storage medium storing a computer programadapted to cause a computer to execute: a procedure of acquiring thecontour of the image of the original from the image of the originalpicked up by shooting the original and acquiring the shape of the imageof the original from the acquired contour; a procedure of determiningthe projection parameters indicating the relationship between the shapeof the image of the original and the actual shape of the original fromthe acquired shape of the image of the original; a procedure ofacquiring the image effect correction parameters for correcting theimage effect from the shape of the image of the original acquired by theshape acquiring procedure; and a procedure of processing the imageeffect for the image of the original having the acquired shape by usingthe acquired image effect correction parameters and performing an imagetransformation of the image of the original by using the acquiredprojection parameters.